Best practices for masters and relays deployed on Linux

The most common cause for deteriorated performance (with frequent crash) of masters and relays on Linux is because the default values set for descriptors are very low. The following configuration checks help you improve the performance (and avoid crashes) of masters and relays on a Linux platform:

Before you begin

Ensure that you have applied the Linux OS updates.

Common OS updates are often applied with the following command lines:  
- RHEL: $ sudo yum update 
- Debian/Ubuntu: $ sudo apt-get update && sudo apt-get dist-upgrade

As of 12.9, it is not mandatory to have all the latest updates. The fontconfig is important. For more information, see this Knowledge Article (KA): Client Management: Graphs are not generated in reports.

To improve the performance of masters and relays on a Linux platform

1. Use Tuned to improve the performance. Tuned is a profile-based system tuning tool that uses the udev device manager to monitor connected devices and enables both static and dynamic tuning of system settings. Tuned is also available for Debian and Ubuntu distributions and it drastically improves the performance.

   The tuned-adm command requires the tuned package (if not already installed). To install the package, run the following command.

   # yum install -y tuned

   A tuning profile consists of a list of system changes corresponding to a specific requirement. To get the list of the available tuning profiles, run the following command.

   # tuned-adm list

   The following profiles are available:

   - latency-performance
   - network-latency
   - network-throughput
   - powersave
   - throughput-performance     
   - virtual-guest

   The throughput performance profile disables tuned and ktune power saving mechanisms, enables sysctl settings that improve the throughput performance of your disk and network I/O, and switches to the deadline scheduler. The CPU governor is set to the performance mode.

   network-throughput is the the recommended profile. It includes the throughput-performance profile and increases network stack buffer sizes.

2. To create a custom tuned profile for BCM, create the bcm-performance directory at /etc/tuned and run the following command.

   # mkdir /etc/tuned/bcm-performance

3. Create the tuned.conf file at the bcm-performance directory and use throughput-performance as the basis and add additional tuning requirements.

   # vi /etc/tuned/bcm-performance/tuned.conf

4. Paste the following lines in the tuned.conf file.

   [main]
   include=network-throughput
   [sysctl]
   fs.file-max = 209708
   net.ipv4.tcp_max_syn_backlog = 5000
   net.ipv4.tcp_tw_recycle = 0

5. Save the file (:wq) and run the following command to validate the changes and then enable the new profile.

   # tuned-adm profile bcm-performance

6. Restart the tuned service:

   # systemctl restart tuned

7. To check new configuration, check the /var/log/tuned/tuned.log file.

   After a successful deployment of this profile, you need to set Linux ulimit settings.

8. Configure the maximum number of open files.

   Linux ulimits
   Tuned does not set Linux ulimit settings, the default limits are almost always too low. In most cases, a limit of 64,000 max user processes and 64,000 open files (both have defaults of 1024) suffice. You can deploy these limits by adding them in /etc/security/limits.conf.

   # vi /etc/security/ limits.conf
   root       soft       nproc     64000
   root       hard       nproc     64000
   root       soft       nofile    64000
   root       hard       nofile    64000

9. Save the file (:wq), reboot the device and run the following commands to validate the changes:

   ulimit -Hn
   ulimit -Sn

   The value returned for each command must be 64000.

10. Edit both /etc/systemd/system.conf and /etc/systemd/user.conf in order to add the following line under the [Manager] section:

    DefaultLimitNOFILE=65535

11. Save the files (:wq).

12. Filesystem type and mount options: We recommend using dedicated disk for BCM master and relay and the EXT4 filesystem with option noatime, nodiratime barrier=0 or XFS filesystem with option noatime. This needs to be done manually in /etc/fstab. You have to set the recommended values at least on the partition where the master or the relay is installed.

    Example for ext4:

    # vi /etc/fstab
    UUID=c49aabe6-8889-4f27-98e7-c155edaaf6ed /       ext4    noatime, nodiratime barrier=0 1 2

    Example for XFS:

    # vi /etc/fstab
    UUID=c49aabe6-8889-4f27-98e7-c155edaaf6ed /       xfs    noatime 1 2

13. For performance reasons in a WAN environment, set the following parameters for a master or a relay and clients.

    mtxagent.ini set:
    DefaultTimeout=30
    DefaultConnectTimeout=30
    ConnectionQueueMaxSize=50
    TunnelTimeout=120

    If you have any questions, please contact BMC customer support.