UNIX

Overview

The UPSTREAM Client on UNIX/Linux is BMC’s software that accepts remote requests to perform data protection including backups, restores, and other operations. The software is typically installed as a daemon running as root and should be running 24/7 in order to receive remote requests. The remainder of this chapter illustrates how to use, and administer the UPSTREAM Client on UNIX/Linux.

Supplemental Information

Supplemental information is included for:

• Command Line UPSTREAM “uscmd”
  
• Hangs in UNIX
  
• Supported Features
  

Managing the UPSTREAM Client on UNIX/Linux

Starting and Stopping the UPSTREAM Daemon

The UPSTREAM Client provides a way to have all of its functions monitored and managed. The usdaemon program is intended to be incorporated into the system startup and shutdown process so that UPSTREAM functions are continuously available while the system is active. If the Client terminates abnormally, usdaemon detects the outage and restarts the Client. An explicit “kill” or “stop” allows UPSTREAM termination and no restart occurs.

The usdaemon restarts UPSTREAM after a three minute waiting period. After three unsuccessful restarts, usdaemon stops restarting UPSTREAM and exits. usdaemon logs UPSTREAM restarts in the SYSLOG file as WARNING messages (see /etc/syslog.conf file for the actual name and location of the “warning” log file).

To manually start the use the fdrupstream daemon program as described in chapter 4.

• ./fdrupstream start
  

To manually stop the usdaemon:

• ./fdrupstream stop
  

UPSTREAM Client Clearing the Log File

The UPSTREAM Client maintains a record of all of it’s activity in a log file called the upstream.log. By default, starting in version 04.01.00, the log files kept forever and never cleared. In order to prevent the Client log from getting too big, the Client provides three ways to clear the upstream.log. In effect, this log clear prunes older entries while preserving new entries. Log clearing can be accomplished:

1. In each UPSTREAM Client. For each Client, the Keep Log for (Days) value in the Advanced screen can be set. Once that day in the log is reached, the Client automatically prunes the log file in one of two cases.
   
   • When the Client is restarted, pruning takes place.
     
   • When the Client registers with the Storage Server. By default this occurs every 24 hours, although this too can be changed.
     
2. During each backup and action. When each UPSTREAM Client backs up a target that backup can be programmed to prune the log at backup end. Use the Client parameter MAXLOGDAYS to prune the log to that number of days. MAXLOGDAYS of 30 means only keep 30 days worth of information.
   
3. Using the uslogclr program. Provided with each UPSTREAM Client on all platforms the program uslogclr clears the log files on demand. Use the uslogclr program on not just the UPSTREAM.log, but any log file the Client writes (including the us.rpt file). Following figure clears out the last 30 days of the log.
   
   

UPSTREAM Client Backups and Restores

Backups - Full System Backups#

Backing up a full system requires careful planning. UPSTREAM is intended for application and user backup. UPSTREAM is NOT a replacement for your operating system backup except in systems where the UPSTREAM Rescuer is supported. We recommend that you continue to use whatever backup mechanism you are currently using to create boot-able backups of the /(root), /usr, /var, /tmp, /opt, and any other directories that are part of /home required for minimal system recovery. A complete /* restore of files from any of the system directories may make your system unbootable. We recommend that only experienced UNIX system administrators perform selected file restores on an as-needed basis (i.e. to recover a deleted /etc/hosts file). However, UPSTREAM is intended to be used as your primary backup facility. Its limitations are predominantly in the area of system disaster recovery. See the UPSTREAM Rescuer chapter for details on your operating system.

Backups - File System Support

The UPSTREAM Client supports the vast majority of file systems found in most installations. Any POSIX compliant file system will be included in the backup, with the exception of ones we know contain no valid data between boots. For example, the /proc file system is excluded automatically on all platforms. If you are interested in the full list of excluded file systems for your operating environment, contact BMC Support .

Backups - Detection of Changed Files

The methodology used to perform backups and detect changed files between Windows and UNIX/Linux is significant, and for day to day backups, how you choose to initiate backups can be important.

As discussed earlier, the UPSTREAM Client on UNIX/Linux can receive a “remote initiate” or a command from a remote system to start an operation. Typically, that operation is a backup but it can include a restore, log request, status request, and other commands. The two most important are backup and restore. In order to keep track of the files on a UNIX / Linux between full and incremental backups, the Client uses one of two methods to track changed files.

1. In the default method to backup files, called “Default”, the Client writes a single line file (<Backup Profile>.inc) that contains the date and time of the backup. When the next incremental is initiated, UPSTREAM uses the date and time in the file and compares it against the last modification date and time of the selected files. While simple and not resource intensive, this method is prone to missed files and other problems.
2. The Client has a second method to detect changed files called the Incremental Database (INCRDB) method. Using INCRDB, UPSTREAM maintains a database of files it has backed up that is used to determine the files to be included in an incremental backup or phase 1 of a merge backup.
   
   

Using INCRDB#

To enable the Incremental Database, set the INCRDB parameter to Y causing UPSTREAM to begin using the database. INCRDB can only be set to Y for the first time for a first-time full or a full merge. To allow easy migration to this method when the option is set for a full merge and there is no existing database, the incremental file is used to determine the files to back up and the database is created.

You can enable this option in the Director in both the Backup and Restore tabs. Press the Options button, and the Miscellaneous tab. Check the Use Incremental Database Facility check-box to enable INCRDB.

The INCRDB option must be set to “Y” for all backups and restores using this profile so that the files are properly recorded in the database. The files are also written to the database when the files are restored so they are not included in the next backup.

The database is stored in the UPSTREAM work path with the file name of the backup profile with an “.idb” extension. This file must be preserved between backups and can potentially grow quite large so sufficient space must be available in the work path.

By default, files that are deleted are detected during incremental backups and noted on the UPSTREAM Storage Server. This results in more efficient and accurate restores from incremental backups. You can turn this option off using the parameter INCRDBDELETEDFILES N.

 Sparse Files

In the UNIX / Linux world a special type of file called a “sparse file” is used to more efficiently store data. The size of this file can be very misleading because when looked at using standard UNIX/Linux commands, like “ls”, the file appears unusually large. In reality, the real data on these types of files is much less than the size the OS returns using “ls”. Of 1.2GB, it is likely that the real data is only a few megabytes. The remainder are zeros.

Prior versions of the UPSTREAM Client reported the inflated size to the UPSTREAM Storage Server causing file allocation requirements to be larger than they should have been. Now UPSTREAM only backs up the “real” data; the smaller size. When a sparse file is restored, UPSTREAM knows that it is a sparse file and restores it as such.

The UPSTREAM Client automatically detects and handles the backup and restore of sparse files without taking up unnecessary space and CPU during the backup or restore.

Logical and Physical Volumes

Logical and physical character volumes as well as files and directories can be backed up and restored. Physical Volume Backup are particularly useful with snapshots, such as Oracle databases. Logical and Physical volumes are not supported in z/OS UNIX Systems Services and Linux OS on IBMZ (they are supported in Linux, but only using the physical backup and restore facility, not using the /dev entry in a regular file backup). To specify a backup of a logical or physical character volume, simply specify the UNIX path of the character special device (for example, /dev/rlv01). You must not have subdirectories selected. You cannot specify wild-cards and these devices do not appear in the Files Available list box to avoid inadvertently selecting them. You must be the root user to use this feature.

When logical or physical volumes are displayed on an inquiry, the device is listed as either <CHR-PV> for a character special physical volume or <CHR-LV> for a character special logical volume along with a short size. Restores of these volumes proceed as for restores of files.

Owners and Groups

UPSTREAM uses the name of the owner user and group for files and directories by default rather than the ID number. This facilitates cross-system restores in situations where the /etc/passwd and /etc/group files on the two systems are not in-sync.

If these files are not in-sync, restores work correctly if the system that owns the disk is the system that is doing the restore. If you are restoring to a disk owned by another system (via NFS or using a vgexport for example) and the two systems do not have user and group ID numbers in sync, the owner and groups assigned may not be as you would expect. Thus we recommend keeping these ID synchronized or using NIS.

The rules for owner/group assignments are:

• The owner is only restored if the restore is performed by root. Otherwise, the owner of the file is the person that performs the restore.
  
• The group is only restored if the restore is performed by root or if the user that performs the restore is the original owner of the file or directory.
  
• If the owner/group name was not found at backup time, UPSTREAM in previous versions used the current UPSTREAM user (root); now the UID/GID from the backup is used.
  
• Existing directory ownership is preserved only if the repeating parameter CHANGEDIRATTRIBS is set to “Y”.
  

Backups include the file or directory’s UID and GID number as well as the UID and GID name. You can force the restore to use the number rather than the name with the restore parameters USEUID and USEGID (defaults to “N”).

If a user name or group name is not available for a UID or GID, UPSTREAM reports errors. You can suppress these errors by specifying the environment variable USNOUIDGIDERRORS=Y; you can suppress obtaining the UID/GID with the repeated parameter NOUIDGIDNAMES=Y (which may also improve performance).

NFS

Your backup spec can be NFS mounted. In the UNIX character mode interface, if you specify an NFS file spec for backups or migrations, there is a display-only check-box NFS Spec that indicates if you specified an NFS file spec (checked) or a local file system (unchecked). File transfers automatically accept NFS mounted files.

If you wish UPSTREAM to traverse NFS mounts while it is traversing subdirectories, set the file spec parameters NFSBELOW Y. For UNIX character mode, there is a check-box Include NFS that is grayed unless you check Include Subdirs. The default is not checked preserving the previous behavior of only backing up local file systems. If your file spec is an NFS spec, then NFSBELOW is automatically set to Y.

Version inquiry details and client reports indicate whether a file spec was an NFS spec or had Include NFS enabled.

Single File System

You can restrict your backups or restores to the file system specified in the backup spec by use of the file spec parameter SINGLEFS Y. This is regardless of where the file system is mounted in the directory structure.

For example, if you have a file system mounted at /home with subdirectories, but then also have separate file system for user ed mounted at /home/ed, if you back up /home with SINGLEFS Y, the /home/ed directory is excluded.

Automatic Snapshot Support (Linux Only)

If you are using Intel or z/Linux, UPSTREAM can automatically attempt to perform a snapshot of the data to be backed up. If the file system is btrfs (which is the default in SuSE12) a file system snapshot is performed. If the file system exists inside of LVM, then an LVM snapshot will be performed. Otherwise, a message will be logged and the backup will proceed by backing up the data directly. If you wish to attempt to do a snapshot, you can specify the non-repeating parameter SNAPSHOT Y or specify the option in the Director.

If the snapshot is an LVM snapshot, then some additional free space will be required in the volume group to handle the snapshot (it’s file changes). This is specified with the parameter SNAPSIZE which defaults to 10% of the size of the file system. If the file system is more active, you may need to increase the size of this value by specifying a larger percentage or the size in bytes, ‘k’, ‘m’, ‘g’ or ‘t’. When preparing new file systems for use with UPSTREAM, if using LVM for snapshots, you should leave sufficient space for the snapshot to run.

After a backup completes, it’s snapshots are automatically deleted. However, if the backup fails and it is restartable, it will automatically remain unless the restarted backup is restart and completes, the restart is deleted in the Director or 7 or more days after the backup starts pass. You can modify this number of days, by specifying the environment variable USSNAPDELETEDAYS.

Loopback File System (Solaris)

A Loopback File System (LOFS) is a virtual file system that provides an alternate path to an existing file system. When other file systems are mounted onto an LOFS file system, the original file system does not change.

UPSTREAM skips the LOFS file system by default to prevent backing up data twice. If you need to backed it up, you can specify “LOFS Y” to force UPSTREAM to include all files in the LOFS in the backup for this particular backup specification.

UNIX Summary

• UPSTREAM can hold open user’s home directories so that they are available for full system backup; specify HOLDUSERDIRS Y. This is particularly useful when performing /* backups for UPSTREAM z⁄OS UNIX where user’s file systems are managed in separate HFS or zFS file systems and are mounted by the system auto-mounter. This parameter is available in the UPSTREAM Director.
  
• For UPSTREAM z⁄OS UNIX. UPSTREAM uses the opendir2/readdir2 API calls that improve performance dramatically in performing directory listings internally. However, the use of these calls may cause zFS file systems to be dismounted or cause other zFS problems. You can cause UPSTREAM to use the lower performance opendir/readdir functions by specifying the environment variable USNOOPENDIR2=Y in your UPSTREAM startup script, the usenv.dat file or your backup/restore job. See UPSTREAM Parameters for the description for setting environment variables.
  
• Significant UPSTREAM internal files, used both by “us” and “uscmd”, including the help file “us.hlp”, the resource file “us.res”, the message file (unless fully qualified in the UPSTREAM configuration), the personalization file “us.ser” and the personalization authorization file “serial.dat” are searched for starting with the default directory (the directory that you were in when you started UPSTREAM), the WORKPATH (in some cases) then in the directory pointed to in the environment variable UPSTREAMPATH.
  
• UPSTREAM backs up device files and other system files when they are included in a backup. By default these files are not included in restores, unless the parameter DISASTERRECOVERY is set to Y.
  
• Symbolic and hard links are properly handled for both backups and restores. See Section “Hard Links” for a description of hard link handling.
  
• Incremental backups are performed by checking the file’s last modification date by default. See“Using INCRDB” in Section.
  
• There is no user specifiable non-file data except for Linux and UPSTREAM for z⁄OS UNIX ACLs. Owner IDs, access dates, and more are stored automatically.
  
• ACLs (and extended attributes) are supported in Linux and UPSTREAM for z⁄OS UNIX only.
  
• If there are errors in starting a remotely requested job (see Advanced UPSTREAM), these errors are stored in the file usjob.out in the WORKPATH directory.
  
• UPSTREAM does not back up directories that have been mounted over.
  
• By default, you cannot restore files to their original locations if the file system has changed. This keeps you from accidentally restoring to an unexpected file system. You can override the restriction by specifying a destination or setting the repeated parameter RESTORETODIFFFS to Y (also found in the Director).
  
• UPSTREAM has a kill and control-C signal handler. If UPSTREAM is terminated with a standard kill or control-C it finishes the current operation and logs message PC1272E External kill. A “kill -9” cannot be intercepted and thus the message is not logged.
  

Supplemental Information#

Command Line UPSTREAM “uscmd”

The command line version of UPSTREAM “uscmd” gives you several advantages over the full screen version:

• It is automatically used as the daemon.
  
• Can be placed in the background.
  
• UPSTREAM can be run using cron.
  

The command line version writes errors and reporting information (if enabled) to stdout. Since this information is also written to the log and report files, you can redirect its output to the nul file to avoid duplication.

The command line version must be started with enough parameters to perform a function unattended as there are no displays to request parameters and stdin is not supported. Thus, we always recommend that you start uscmd with a command line specified parameter file:

uscmd parameter=<parameter file>For example, to process a backup request stored in an UPSTREAM parameter file “testback.dat”, you would enter:

The “uscmd” child process (uscmd1, uscmd51, uscmd280, etc. depending upon the operating system) can be killed in the normal way and terminates whenever the requested function has completed.

Hangs in UNIX#

When performing an UPSTREAM backup of a UNIX machine and the backup appears to hang, there are several questions to ask that may help figure out the cause of the hang:

1. Is the PID still active on the system? In the upstream.log we log the PID that the UNIX machine assigns to the backup/restore. After a backup hangs and there are no more messages written to the upstream.log, do a “ps ­ef|grep uscmd” to see if that PID is still active.
2. Do a “kill” (not -9) of the PID. This might allow the process to finish writing to the upstream.log report the file it was hung on.
3. If that does not stop the process, you need to do a “kill ­9”. Most likely, this does not write any further information to the log or report file but allows the process to terminate. If it does not, it is definitely locked in a file system call.
4. Did the backup write any files to the host, or did it hang while building the backup file? You can determine this by looking at the host log. If it hung during the build phase, then issue the following command to see where it might be hanging: “find /”.
5. If it hung after files were sent to the host, then the report file may show where it is hanging. But if UPSTREAM was killed without cleaning itself up, UPSTREAM writes a temporary report file, named us.<pid>.rpt. We recommend using REPORTOPTIONS containing at least the value 35.
6. Is the backup target a BCV? BCV's can go “not ready” and cause hangs.
   NFS? NFS mounts can go stale and result in hangs.
   
7. Are there millions of files? It might simply be that slow.
8. Does a directory listing take forever to return? If yes, increase the global time-out value on the UPSTREAM host.

If it is hung in a file system call, exclude the offending files or use the environment variable, USSEARCHSKIP. If it is hung in a TCP/IP call, the “kill” indicates an interrupted system call.

Supported Features#

The UPSTREAM Client supports the following UNIX/Linux features.

• Restore to New Mount Points
  
• By default, you cannot restore files to their original locations if the file system has changed. This keeps you from accidentally restoring to an unexpected file system. You can override the restriction by specifying a destination or setting the repeated parameter RESTORETODIFFFS to Y.
  
• External Links
  
• Symbolic and Hard Links
  
• Symbolic and Hard links are properly handled for both backups and restores. See Section “Hard Links” for a description of hard link handling.
  
• Auditing flags
  
• HFS/zFS Extended Attributes
  
• z⁄OS UNIX tags. Tags are automatically restored with a file. If you wish to have tags removed from existing files as well, set the environment variable USDELETEBEFORERESTORE to Y (in usenv.dat or in your job).
  
• UNIX owners and permissions
  
• Case sensitivity
  
• FIFOs
  

• ACL’s (Linux and z/OS UNIX only)