Performance tuning parameters

The following sets of parameters were configured to achieve optimal performance during BMC benchmarking tests with BMC Cloud Lifecycle Management. You might need to further tune these parameters value to achieve optimal performance under different workload and infrastructure conditions.

JVM tuning parameters for all components

The following table lists the recommended values for various JVM parameters with different BMC Cloud Lifecycle Management components that were used in the BMC performance lab.

Platform Manager tuning

Use the following parameters with the cloudservices.json and providers.json files (located by default at C:\Program Files\BMC Software\BMCCloudLifeCycleManagement\Platform_Manager\configuration).

cloudservices.json

The Maximum Connections and Maximum number of connections parameters are the connection parameters for Session Manager and the Cloud database service. Session Manager manages the user sessions and keeps track of user and task sessions. It keeps session info in an AR System form; to connect to that AR System form, it uses these connections. Maximum number of connections is used by Cloud database service for performing CRUD operations on the CMDB.
Keep both these connection parameter values in sync. The recommended values are:

• Maximum Connections = 50
• Maximum number of connections = 50

Task Manager is a distributed service that is available as a core service in all containers. It is responsible for executing tasks in an efficient, fault tolerant, and scalable manner.  Any operation on a resource that is not a strict read (for example, a GET-call) will be implemented in an asynchronous fashion. These asynchronous operations are modeled as tasks in the Platform Manager.

You can define the following Number of task manager threads and Max number of task manager threads:

• Number of task manager threads: Number of threads that Task manager starts with. These threads will be running in the system and serving events (these are internal to Task managers). The events are responsible for rolling the states within the FSM.
• Maximum number of task manager threads: The maximum number of task manager threads in the system. This number also denotes the maximum number of tasks running in PARALLEL in the system. If there are more tasks in the system than specified by this number, they wait until one of the tasks finishes or hibernates and then frees up the thread.

The recommended values are:

• Number of task manager threads = 100
• Maximum number of task manager threads = 300

providers.json

Virtual Machine cloning time on the vCenter depends on the underlying hardware, network connectivity, and Configurations Max parameters from VMWare. To accommodate higher cloning timing under bulk load, you must tune certain parameters on Platform Manager, BMC Server Automation, and the vCenter. The following lists some of the most important parameters in the Providers.json file:

• BBSA_VG_JOB_STATUS_POLLS indicates the total number of outgoing BSA API calls (from CSM) before the Virtual-Guest constructor operation times out in CSM, leading to SOI provisioning timeout. The value for this parameter is recommended as: 

  "name" : "BBSA_VG_JOB_STATUS_POLLS " }, "attributeValue" : "240",

• BBSA_VG_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_VG_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as: 

  "name":BBSA_VG_JOB_STATUS_POLL_INTERVAL_SECONDS"}, "attributeValue" : "60",

• BBSA_USP_JOB_STATUS_POLLSindicates Create user and password in VM or Physical Server provisioning: Update Server Properties Job status polls – total number of outgoing BSA API calls (from CSM) to determine the USP Job status before the operation times out in CSM leading to SOI provisioning timeout. The value for this parameter is recommended as:

  "name" : BBSA_USP_JOB_STATUS_POLLS " }, "attributeValue" : "240"

• BBSA_USP_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_USP_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as:

  "name" : BBSA_USP_JOB_STATUS_POLL_INTERVAL_SECONDS " }, "attributeValue" : "60"

• BBSA_SCRIPT_JOB_STATUS_POLLS indicates Opening Ports (execute Script) and so on in a provisioned VM: Execute Script NSH Script Job status polls – total number of outgoing BSA API calls (from CSM) to determine the USP Job status before the operation times out in CSM leading to SOI provisioning timeout. The value for this parameter is recommended as:

  "name" : "BBSA_SCRIPT_JOB_STATUS_POLLS" }, "attributeValue" : "240"

• BBSA_SCRIPT_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_SCRIPT_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as:

  "name":"BBSA_SCRIPT_JOB_STATUS_POLL_INTERVAL_SECONDS"}, "attributeValue" : "60"

• BBSA_SOFTWARE_JOB_STATUS_POLLS indicates Installing Software (Apache, and so on): The total number of outgoing BSA API calls (from CSM) before the Software Resource constructor operation is timed out in CSM BBSA provider, leading to CSM Software deploy to fail. The value for this parameter is recommended as: 

  "name":BBSA_SOFTWARE_JOB_STATUS_POLLS" },"attributeValue": "240"

• BBSA_SOFTWARE_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_SOFTWARE_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as: 

  "name":BBSA_SOFTWARE_JOB_STATUS_POLL_INTERVAL_SECONDS"}, "attributeValue" : "60"

• BBSA_VG_AGENT_STATUS_POLLS indicates the status of the BBSA Agent on the VM:  The total number of outgoing BSA API calls (from CSM) to determine the agent status on the provisioned VM before the Virtual-Guest constructor operation times out in CSM, leading to SOI provisioning timeout. The value for this parameter is recommended as: 

  "name" : "BBSA_VG_AGENT_STATUS_POLLS" }, "attributeValue" : "240"

• BBSA_VG_AGENT_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_VG_AGENT_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as: 

  "name":"BBSA_VG_AGENT_STATUS_POLL_INTERVAL_SECONDS" }, "attributeValue" : "60"

BMC Server Automation server tuning

BMC Server Automation is the back-end component of the cloud solution that is used for provisioning VMs and setting them up for usage. BMC Cloud Lifecycle Management uses Virtual Guest Jobs (VGJs) to create VMs and then later runs USP (Update server properties) NSH scripts to fully configure the VMs. It can additionally involve running software deploy jobs.

Use the following set of recommendations to configure and set up the BMC Server Automation application server and job servers for getting a through-put of 250 VMs/hr with a single vCenter.

Assumptions

• These recommendations assume a through-put of 250VMs/hr using a single vCenter. You can extrapolate any additional throughput requirement, using additional vCenters and job servers.
• The vCenter is configured so that it can clone and customize a single VM within ~10-12 minutes.
• The recommendations are for creation of VMs and setting them up; any additional load on BMC Server Automation must be considered separately and accounted for.

Recommendations

• Use a single Configuration Server with multiple Job Servers to handle various jobs.
• Configuration Server: Not used as a job server
• Job Server for VGJs, USP, and Create user jobs (Through-put: 250 jobs/hr - each) - 3 Numbers
  Use the blasadmin -s option.
  
  • MaxJobs: 75 (per job server)
    For example: 
    blasadmin -s <job_deployment_name> set appserver MaxJobs 75
  • MaxWorkItemThreads: 100 (per job server)
    For example: 
    blasadmin -s <job_deployment_name> set appserver MaxWorkItemThreads 100

• If BSA Job servers are used for running any other additional types of jobs, then dedicate the previous two job servers for the VGJs, USP, and NSH jobs, by setting affinity rules. You can define affinity rules by using following key words:
  • "VG_Job"
  • "USP_Job"
  • "create_user_job"
• For all other jobs, configure separate jobs servers depending on the load, using standard BMC Server Automation sizing procedures.
• Use the blasadmin utility to change the following timeout values for all the job servers, configuration server, and scheduler. Use the blasadmin -a option.
  • BlTargetJobManagerTimeout: 14400 (seconds)
    For example: 
    blasadmin -a set appserver BlTargetJobManagerTimeout 14400
  • MaxJobTimeInSchedulerQ: 300
    For example:
    blasadmin -a set scheduler MaxJobTimeInSchedulerQ 300

Recommendations for Compliance

For details about best practices for running Compliance Content templates, see Compliance Content support and requirements in the BMC Server Automation documentation.

BMC Server Automation agent

Increase the number of threads for COServer on the vCenter Server BSA agent by editing the AssetImplConfig.xml file. This file is located at: 
<AgentHomeDirectory>\8.1\RSCD\daal\Implementation\BMC_VMware_VirtualInfrastructureManager_win64\ win64\AssetImplConfig.xml

• assetImpl_coServerMaxThreadCount = 100
• assetImpl_coServerFragmentSize = 16384
• assetImpl_coServerReceiveWaitTime = 30
• assetImpl_coServerResponseTimeoutTime = 3600
• assetImpl_cacheTimeout = 3600

Change the following parameters listed in the vmware.properties file. This file is located at:
%install directory%\BMC Software\BladeLogic\RSCD\daal\Implementation\BMC_VMware_VirtualInfrastructureManager_win64\win64.

• com.bladelogic.vmware.task.timeout= 18000000 (milli seconds)
• clone-timeout= 18000000 (milli seconds)
• customization-timeout= 18000000 (milli seconds)
• customization-timeout-win2k8= 18000000 (milli seconds)

BMC Server Automation agent (Hyper-V)

Increase the number of threads for COServer on the Hyper-V Server BSA agent by editing the AssetImplConfig.xml file. This file is located at:
<AgentHomeDirectory>\RSCD\daal\Implementation\BMC_SCVMM_win64\win64\AssetImplConfig.xml
Use the BSA Agent values previously listed.

In addition, modify the values of the following parameters in the scvmm.conf file. This file is located at: 
<AgentHomeDirectory>\RSCD\daal\Implementation\BMC_SCVMM_win64\win64: 

• TemplateJobRetryCount=100
• InitialWaitForWin2K8Sec=240

Cloud Portal and Database server tuning

The following table lists the recommended parameters in ar.cfg or ar.conf to configure the Cloud Portal and Database server.

Mid-Tier tuning

The following table lists the recommended values in <Apache_Install_Dir>\Tomcat6.0\conf\server.xml to tune the web infrastructure that hosts the Mid-Tier (the Tomcat servlet).

TrueSight Operations Management tuning

For BMC ProactiveNet tuning, see: https://docs.bmc.com/docs/display/public/tsim10/Sizing+and+scalability+considerations+and+best+practices.

BMC ProactiveNet tuning

For BMC ProactiveNet tuning, see https://docs.bmc.com/docs/display/public/proactivenet96/Performance+and+scalability+recommendations.

Database tuning

Use the following parameters to tune your SQL Server or Oracle database.

MS SQL Server

Based on performance tests conducted in the performance lab, BMC recommends the following Microsoft SQL Server tuning practices:

• Set PARAMETERIZATION database option to FORCED.
• Set READ_COMMITTED_SNAPSHOT database option to ON.

• Set ALLOW_SNAPSHOT_ISOLATION database option to ON.

  Note

  To set ALLOW_SNAPSHOT_ISOLATION and READ_COMMITTED_SNAPSHOT to ON, there must be no active connections to the database except the connection that is executing the ALTER DATABASE statement. When you finish turning on these two database options, multiple connections are supported.
• Update the statistics regularly by using sp_updatestats stored procedure on SQL Server.
  sp_updatestats updates the query optimization statistics on a table or indexed view. By default, the query optimizer already updates statistics as necessary to improve the query plan. In some cases, you can improve query performance by using UPDATE STATISTICS or the sp_updatestats stored procedure to update statistics more frequently than the default updates.

• Use the Maximum Degree of Parallelism configuration option (MAXDOP).
  When you configure your SQL Server database server, set the Maximum Degree of Parallelism option to limit the number of processors used in parallel plan execution. By default, this option is set to 0, which uses all available processors. The problem is that one long-running, CPU-intensive SQL statement could monopolize all processors and create long wait times for other users. If your transaction environment shows long-running, CPU-intensive SQL statements, set the max degree of parallelism (MDOP) as follows:

  Number of cores or CPUs on SQL server	MDOP setting
  1–7	0 (default)
  8–15	6
  16 or more	8

Oracle

When tuning Oracle database server, use the following checklist:

• Make sure that the alert.log is clean.
• Use an appropriately sized SGA.
• Set cursor_sharing to FORCE.
• Set Processes value to 500+ for the Enterprise-AR database instance.
• Run gather stats.
• Use AWR reports to collect diagnostics data. Review the top-timed events for potential bottlenecks.
• Use an AWR report to monitor the database for the top SQL statement. Make sure that the SQL execution plans are optimal.

Docker host tuning

To provision 250 or more Docker containers, perform the following steps on the Docker host:

1. Increase the default value (1024) of the open files parameter (ulimit -n) to 2048 or higher.
2. Start the Docker daemon.

Overall system tuning

This section provides guidelines for optimizing the network, client configuration, and overall system.

Performance considerations

Remember the following performance considerations:

• Create a dedicated environment for the BMC Cloud Lifecycle Management products. Performance is difficult to manage when multiple applications are running in the same environment.
• Sizing the database server for capacity is important, because it may not be possible to scale the database tier horizontally unless you are using Oracle RAC.
• Database performance is highly dependent on a fast I/O subsystem. You must have an optimum I/O subsystem.
• Since the AR System server only caches metadata, it is highly dependent on the fast, reliable, low-latency connection to the database.
• Use a gigabit connection with close to zero latency between the AR System server and the database. Any additional hop, packet screener, or firewall has a negative impact on performance.
• On virtualized environments, the vCPU to pCPU ration is workload dependent. For optimized performance, BMC recommends 1:1 vCPU to pCPU ratio.

Optimizing the network

To optimize the network, BMC recommends the following:

• Do not install a firewall between the AR System server and the database server, because the firewall has a significant negative impact on the performance.
• Use a gigabit network with minimal latency between the AR System server and the database server.
• If there is a load-balancer or firewall between the Mid-Tier and the AR System server, configure it not to sever idle connections from the Mid-Tier. Reestablishing severed connections takes additional time.

Optimum client configuration 

The following lists the optimum client/browser configuration:

• Dual Core 2+ GHz CPU with 2 MB L2 cache or single core 3+ GHz CPU with 2 MB L2 cache
• 2-4 GB RAM
• Firefox 3.6 (or later) or Internet Explorer 8 (or later)
• Disable all browser add-on toolbars and plug-ins except Adobe Flash.
• Configure the browser to open pop-ups in a new tab.
• Configure the browser cache setting to Automatic.

Best practices for tuning the overall system

Use the following checklist when tuning your overall system:

• Survey each area of your system for possible tuning issues before you make any changes.
• Prioritize all changes by their likely impact on the identified problem.
• Implement changes one at a time in a prioritized manner.
• Quantitatively evaluate the impact of every change before you consider whether another change should be made.

Performance tuning checklist

The following table shows the performance tuning checklist for all the BMC Cloud Lifecycle Management components.