Performance

Overview

High performance is one of the major features of UPSTREAM. But, as with any communications product, performance requires some tuning to adjust it for your environment.

You must scientifically isolate the performance bottlenecks and then take the actions based on where the bottlenecks are. Your goal is to avoid wasting time optimizing areas where performance is not bottle-necked.

UPSTREAM provides performance isolation tools based on where performance bottlenecks generally are.

Performance Tests#

There are a number of performance tests built into UPSTREAM. Each test is described below with performance tuning suggestions. When you are first attempting to isolate performance issues it is recommended that you select all the options and then when fixing each option, perform one fix at a time until acceptable results are obtained. Feel free to contact BMC Support at any step in the process to help you get the best from your environment and to help you set your expectations.

Setting your expectations reasonably is an important step. There is a finite point where you cannot appreciably improve performance. Remember that your goal is to get the best possible performance.

Performance tests are most often performed using the UPSTREAM Director.

Raw Communications Tests

The Raw Communications performance tests are available in the Director in the Target Systems list. To get to the list, press the Target icon or pull down the View menu and select Registration/Target List. From the Target Systems list, highlight an active target, and press the Performance tests button to display the Performance Test dialog:

Performance Test DialogUPSTREAM_Client_Performance_image167.gifimage2021-8-19_9-49-23.png

The raw communications tests are the most used performance tests as they indicate in a simple step, the ability of the communications network to transfer large quantities of data. All users are encouraged to run these tests when tuning UPSTREAM as the initial starting point for isolating performance bottlenecks. We also recommend running these tests whenever setting up a new workstation/server to verify that communications performs as expected.

Check the performance test option you wish, along with the record size and record count, and press the Start button to begin the test.

You are notified with a message box when the test is completed. As with all functions in the Director, If you encounter any errors you can view the remote UPSTREAM messages for the event; pull down the View menu, select the Messages sub-menu and the Current Target item.

Raw Communications Test - PC Sends

This test allows the sending of a specified number of blocks of a specified size to the Storage Server. This test can be used to determine the overall throughput that can be accomplished in a backup. When this box or the Raw Communications Test - PC Receives box is checked, the record size and record count fields become available for entry.

Important

The total number of bytes transmitted is the product of the record size and record count fields. Do not specify values too large as this test cannot be interrupted once begun.

Raw Communications Test - PC Receives

Allows the receiving of a specified number of blocks of a specified size from the Storage Server. This test can be used to determine the overall throughput that can be accomplished in a restore. When this box or the Raw Communications Test - PC Send box is checked, the record size and record count fields become available for entry.

Important

The total number of bytes transmitted is the product of the record size and record count fields. Do not specify values too large as this test cannot be interrupted once begun.

Other Performance Tests#

The primary other performance tests are used to read data only (File I/O test), read data like in a backup and do not send (No Communications I/O) it and read data, send it and do not write it (No Host Disk I/O).

You can access these tests from the Backup Specifications screen. After you complete the parameters necessary for a backup, the Performance test button activates. You can press this button to specify these tests:

Backup Performance TestsUPSTREAM_Client_Performance_image169.gifimage2021-8-19_9-51-36.png

No Communications I/O

This test does all the functions of a backup, but does no data transmission to the Storage Server. This is the most useful test as it shows the entire client side of a backup (read, meta-data access, compression, etc.) without the costs of transmission and remote storage.

See File Read Testfor suggestions for disk tuning based on your results.

Also note that since this test does reads of metadata, there may be an advantage in reducing the amount of metadata you back up (which might include using just UID/GID numbers rather than names for example). If you do choose to reduce the amount of metadata you back up, you should contact BMC Support to make sure that you are not excluding some metadata that may be critical in the type of restores you need to perform.

Note that besides simple disk reads, this test also shows the affect of compression. You may find that reducing compression level will improve performance as it takes less CPU overhead.

Note that the backup profile is changed to $PRFTEST to avoid affecting any local profile information, and certain backup types including most databases and plug-ins are not allowed. The ones allowed are NotesR5, ExchMail, and WinSS if no components are specified.

File Read Test#

This test reports the performance of reading all the files specified in the first backup file specification screen in characters per second. This test does not test subdirectories even if the subdirectory box is checked. The record size is used to determine the blocking of the files read. Note that in many cases a Backup - No Communications performance test is more definitive as it reads all non-file data as well as well as traversing subdirectories.

PC file read speeds are critical in backup performance. In many file server environments, file I/O performance is the principal bottleneck. But these numbers can be deceiving. Most operating systems are usually fast in reading an open file but are slow in the file opens. And as the number of files in a directory increase, the file opens get slower. In the same environment it is common to see speeds ranging from 10,000,000 characters per second to 20,000 characters per second depending on the file sizes and the number of files in the directory. So plan on running this test several times on different directories to get a reasonable estimate of aggregate throughput.

The file read/write size in UPSTREAM is not controlled by the record size, but by the BACKUPBUFFERSIZE environment variable (see “Backup Fine Tuning” in Section for more information). The default is 262144 for most environments.

To improve file read performance there are several things to try:

  • Check real-time virus checking settings. If you have real-time virus scanning enabled on your system, this can severely impact UPSTREAM’s throughput.
  • If you are accessing files via a LAN, look into adapter tuning. For example, with Ethernet LANs you may need to try changing the duplex (full to half or vice-versa), or explicitly setting the adapter speed (to 100Mb for example).
  • If you have many large files, increasing the BACKUPBUFFERSIZE may help to improve performance.
  • In some environments (particularly Solaris servers), reducing the BACKUPBUFFERSIZE has shown in some tests to improve performance.
  • Reorganize your directories to reduce the number of files in a directory. It is faster to open a small number of files in many directories than a large number of files in a single directory.
  • Faster CPUs improve performance. If you can, use a faster workstation/server. Increasing the memory in servers often helps improve performance as well.

Backup - No Host Disk I/O

This test helps you by performing all the functions of a backup without writing the data on the Storage Server. A normal backup is done using the backup specifications currently defined. The data is transmitted to the Storage Server but the Storage Server does not record it. You can not use UPSTREAM/ SOS (local backup disks) with this test.

It is a good idea that before you run this test you perform a File Read or Backup - No Communications test to determine the file read performance to compare against total throughput reported. You should perform Raw Communications tests before running this test to see how backup performance changes when real data is read.

Thus this test is good to helping you in see the affect of latency (during the read) and slightly different buffer sizes (as UPSTREAM record packing sends slightly varying record sizes) in overall performance.

Note that the backup profile is changed to $PRFTEST to avoid affecting any local profile information, and certain backup types including most databases and plug-ins are not allowed. The ones allowed are NotesR5, ExchMail, and WinSS if no components are specified.

TCP/IP Tuning#

TCP/IP is somewhat more difficult to tune as it is designed to be self-tuning. However, there are some things that can be done to improve your performance. Some of these include:

Note that many of tuning parameters suggested here affect all your TCP/IP applications. As a starting point, we recommend that you perform a bulk move with a standard TCP/IP application (such as FTP) to get a benchmark of non-UPSTREAM TCP/IP performance. Then perform the Raw Communications tests using UPSTREAM to see how UPSTREAM performance compares to performance with other TCP/IP applications. As you modify parameters, verify that your changes do not negatively impact performance by comparing them against your initial benchmarks.

LAN Adapter Tuning#

A poorly tuned LAN adapter can degrade performance significantly for all LAN accesses. Making sure that you have the newest driver from the manufacturer is the first step. Then verify that your settings are correct (full vs. half duplex, LAN speed, etc.). Note that these setting must match your other hardware (such as Ethernet switches). For Ethernet, you may want to try changing the duplex even if you are confident that you have duplex matched with your switch.

Send and Receive Buffer Sizes#

The lower layers in TCP/IP allocate buffers that are used to hold send and received data in transit. Increasing the size of these buffers can, in some cases, make a dramatic difference in performance. The initial sizes are vendor dependent.

You can set these buffer sizes in the advanced options in the Configurator: TCP/IP Send Buffer and TCP/IP Receive Buffer.

We have found that most often the value 65535 seems to work the best and is the default for most environments. However:

  • In a number of UNIX environments, the system default (which is set by no value or a value of 0) occasionally works better than 65535.
  • z/OS UNIX Systems Services seems to always work best with a value of 32768 and it is the default in that environment.

You should run Raw Communications performance tests to verify the best value for your environment.

UPSTREAM Record Size

The UPSTREAM transmission record size can, in many cases, be the most significant performance modifier. The data record transmitted in a single TCP/IP send command is determined by the structure of the data but is as close as possible to the Packing Size (the PACKRECSIZE parameter), which is specified in the Backup or Restore More Dialogs.

If you are working with an UPSTREAM Storage Server v3.7 or above you can use a packing size of up to 65400. Using the largest possible value often appears to improve performance and reduce CPU utilization on both sides.

To determine your optimal Packing Size, perform multiple UPSTREAM Raw Communications Performance tests, using various record sizes. We recommend that you try 6000, 8192, 12000, 18000, 24000, and 32700 (the maximum that the performance test supports); do not forget to reduce the record count as you increase the record size to keep the tests from running too long. If you find that your performance is best at smaller record sizes (6000 is much better than 32700), then set your record size to the record size that you had the best performance with and set your packing size to be 0 (to disable packing) or near that size, but slightly larger (say 6200 for example).

Setting TCP/IP parameters#

Some system specifics for setting overall TCP/IP parameters:

Windows

Virtually all TCP/IP parameters can be modified. Parameters are set using manual registry modifications. Microsoft Knowledge Base article Q120642 describes the process in detail. Note that most parameters are properly negotiated.

Others

Most other systems (Windows, many UNIX systems, etc.) provide few if any modifications. See your system administration guides for more information.

Increasing MTU size#

The maximum transmission unit (MTU) is the largest size that a IP packet can be. It is fixed at 1500 for many forms of Ethernet. If you have the opportunity to use “Jumbo Packets”, this almost always result in better performance, particularly with gigabit Ethernet and above where the overhead in processing packets becomes overwhelming.

Window Size#

This parameter determines the maximum TCP receive window size offered by the system. The receive window specifies the number of bytes that a sender may transmit without receiving an acknowledgment. In general, larger receive windows improve performance over high (delay * bandwidth) networks. For highest efficiency, the receive window should be an even multiple of the TCP Maximum Segment Size (MSS).

UPSTREAM Storage Server Tuning

The Storage Server manuals have a number of performance tuning suggestions.

Eliminate the LAN

You can eliminate the LAN by using UPSTREAM/SOS or the SAN Express or some other technique that avoids transmission of the data. If the LAN is your primary bottleneck, this will certainly help.

Other TCP/IP Notes

Some TCP/IP issues you may run into:

  • TCP/IP Time-outs during long tape mount delays: If possible, adjust upwards your Maximum Data Retransmission Count (for Windows, the parameter is TcpMaxDataRetransmissions). Adjusting the data retransmissions value upwards slightly (from 5 to 20 for example) generally gets rid of the problem. See Microsoft Knowledge Base article Q120642 for more information.
    For Windows platforms a simple program to read/set the TCP/IP registry time-out values (TCPTIMEOUT.EXE) helps avoid direct registry editing.
  • Situations where security validation works, but backups or restores do not can often be caused by MTU or Window Sizes set too large.

TCP/IP has a number of architected return codes. Some of their more common ones and their likely causes are:

Broken Pipe

Usually caused by an intermediate system outage such as a lost router, a pulled cable, etc. This can also be caused by the remote system going down (crashing).

Address Already in Use

Usually caused by brining up a 2nd copy of UPSTREAM listening on the same port.

Connection Reset By Peer

This is the most common error and it can be caused by a remote system or remote application crash, or an intermediate software failure (long tape mounts may cause this. See the TCP/IP time-out values and TCPTIMEOUT.EXE above).

Connection Refused

Usually caused by UPSTREAM not running on the remote system.

Other Performance Improvements#

Besides the issues you can isolate and help improve in the performance facility, other performance improving items include:

Local Backups/SAN Express#

Local Backups are a technique of UPSTREAM where some data is stored on the local system as well as being transmitted to the UPSTREAM Storage Server. In low-communications speed environments this technique can significantly improve restore performance. See the Local Backup section for a more detailed description of this facility.

Line Speed#

Line speed is the most important performance modifier. However, for most users this is the hardest to modify. But, when looking for the largest improvements, this is where you must start.

Multiple Clients#

Running more than one client at a time, on the same LAN with the same server provides better performance than a single machine, due to the advantages of using multiple CPUs and multiple conversations.

The key to doing this is careful planning. For example, if you have two volumes on your file server, you may choose to use two PCs, each backing up one volume. Remember that you must use separate backup profiles.

Bridges and Routers#

Bridge and/or router hops reduce overall performance. To increase performance, the goal is to place as few intermediate devices as possible among the file server and the backup PC, and the backup PC and the Storage Server connected device.

Test Times#

Test UPSTREAM when you plan on performing the backup. File server congestion, Storage Server connection congestion, Storage Server utilization and other multi-user usage issues impact performance, and may in fact be less significant at the time the backups run. If you can, attempt to run your final performance benchmarks at the same time of day and the same day of the week the backups plan to be run.

Screen Savers#

Screen savers can take an enormous amount of CPU overhead. It is recommended that screen savers always be disabled.

Server Compression#

Some file server systems support compression. Since UPSTREAM uses standard file access methods, the file server system is constantly working to decompress/recompress files for transmission to the UPSTREAM machine. It is recommended that file server compression be disabled when optimizing for performance. The exception for this is Novell NetWare, where the data is stored compressed.

Record Size Tuning#

The UPSTREAM parameter Record Size is the size of the logical record stored on the Storage Server. However, record size does not have to be the physical disk read/write record size or the communications send/receive record size.

To change the transmission record size, there is a parameter PACKRECSIZE, whose default is 32700 (use with and UPSTREAM Storage Server v3.7 or above allows a packing size of up to 65400), which defines the TCP/IP or SNA transmission size. In most environments the largest value is the best value, however, you may find that you may be able to improve performance by reducing it.

To change the disk read/write size, specify an appropriate Record Size (usually best done as a power of 2 i.e. 4096, 8192, 16384) and then specify the environment variable parameter BACKUPBUFFERSIZE to be a multiple of this value (up to 65500). The data read size is an even multiple of the Record Size up to the BACKUPBUFFERSIZE. The default is 32768.

Careful Use of Compression#

Compression can help or hurt performance. If the time to process data is more than made up by the time you have to send or receive less data, then compression helps. You should try testing all the compression levels on sample data in a realistic environment to determine which one, if any, helps the best.

Issues that affect compression include:

  • CPU performance and load. Low available CPU makes high compression less useful.
  • Available communications bandwidth. High bandwidth links tend to work better with lower level of compression. Low bandwidth links work better with higher level of compression.
  • Data composition. Massively compressible data tends to work better with higher levels of compression; already compressed data can get larger with compression.
  • Target device. If you are writing to tape which compresses the data, the amount of tape can increase and the backup performance degrade with higher levels of compression.

Because of all the variables that affect the efficacy of compression, we strongly recommend communications performance test to verify the communications bandwidth available, and No communications performance tests to try the different levels of compression on your data.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*