Unsupported content

 

This version of the documentation is no longer supported. However, the documentation is available for your convenience. You will not be able to leave comments.

Performance tuning parameters

The following sets of parameters were configured to achieve optimal performance during BMC benchmarking tests with BMC Cloud Lifecycle Management 3.1.x. You might need to further tune these parameters value to achieve optimal performance under different workload and infrastructure conditions.

JVM tuning parameters for all components

The following table lists the recommended values for various JVM parameters with different BMC Cloud Lifecycle Management components that were used in the BMC performance lab.

Name or Function

JVM Parameters

Enterprise-AR

Use default settings

Cloud-AR

Use default settings

Mid-Tier

Use default JVM settings and also add the below parameters:
JVM hybrid mode and GC
-XX: +UseCompressedOops -XX: +UseParallelGC (Remove default GC i.e.
"Xincgc")

BMC Server Automation

Config and Job Servers
MaxHeapsize: 4096 MB

BMC Network Automation

Use default settings

Platform Manager

Use default settings

BMC Atrium Orchestrator

Use default settings

vCenter Server

COServer Plugin (JPAProgramConfig.xml of vCenter Server)
-Xms: 512 M
-Xmx: 2048 M

Note

To run the Day 2 scenarios (Audit/Compliance /Patching …), increase the heap size to 4096 MB for Configuration and Job servers.

Platform Manager tuning

Use the following parameters with the cloudservices.json and providers.json files (located by default at C:\Program Files\BMC Software\BMCCloudLifeCycleManagement\Platform_Manager\configuration).

cloudservices.json

The Maximum Connections and Maximum number of connections parameters are the connection parameters for Session Manager and the Cloud database service. Session Manager manages the user sessions and keeps track of user and task sessions. It keeps session info in an AR System form; to connect to that AR System form, it uses these connections. Maximum number of connections is used by Cloud database service for performing CRUD operations on the CMDB.
Keep both these connection parameter values in sync. The recommended values are:

  • Maximum Connections = 50
  • Maximum number of connections = 50

Task Manager is a distributed service that is available as a core service in all containers. It is responsible for executing tasks in an efficient, fault tolerant, and scalable manner.  Any operation on a resource that is not a strict read (for example, a GET-call) will be implemented in an asynchronous fashion. These asynchronous operations are modeled as tasks in the Platform Manager.

You can define the following Number of task manager threads and Max number of task manager threads:

  • Number of task manager threads: Number of threads that Task manager starts with. These threads will be running in the system and serving events (these are internal to Task managers). The events are responsible for rolling the states within the FSM.
  • Maximum number of task manager threads: The maximum number of task manager threads in the system. This number also denotes the maximum number of tasks running in PARALLEL in the system. If there are more tasks in the system than specified by this number, they wait until one of the tasks finishes or hibernates and then frees up the thread.

The recommended values are:

  • Number of task manager threads = 100
  • Maximum number of task manager threads = 300

providers.json

Virtual Machine cloning time on the vCenter depends on the underlying hardware, network connectivity, and Configurations Max parameters from VMWare. To accommodate higher cloning timing under bulk load, you must tune certain parameters on Platform Manager, BMC Server Automation, and the vCenter. The following lists some of the most important parameters in the Providers.json file:

  • BBSA_VG_JOB_STATUS_POLLS indicates the total number of outgoing BSA API calls (from CSM) before the Virtual-Guest constructor operation times out in CSM, leading to SOI provisioning timeout. The value for this parameter is recommended as: 

    "name" : "BBSA_VG_JOB_STATUS_POLLS " }, "attributeValue" : "240",
  • BBSA_VG_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_VG_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as: 

    "name":BBSA_VG_JOB_STATUS_POLL_INTERVAL_SECONDS"}, "attributeValue" : "60",
  • BBSA_USP_JOB_STATUS_POLLSindicates Create user and password in VM or Physical Server provisioning: Update Server Properties Job status polls – total number of outgoing BSA API calls (from CSM) to determine the USP Job status before the operation times out in CSM leading to SOI provisioning timeout. The value for this parameter is recommended as:

    "name" : BBSA_USP_JOB_STATUS_POLLS " }, "attributeValue" : "240"
  • BBSA_USP_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_USP_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as:

    "name" : BBSA_USP_JOB_STATUS_POLL_INTERVAL_SECONDS " }, "attributeValue" : "60"
  • BBSA_SCRIPT_JOB_STATUS_POLLS indicates Opening Ports (execute Script) and so on in a provisioned VM: Execute Script NSH Script Job status polls – total number of outgoing BSA API calls (from CSM) to determine the USP Job status before the operation times out in CSM leading to SOI provisioning timeout. The value for this parameter is recommended as:

    "name" : "BBSA_SCRIPT_JOB_STATUS_POLLS" }, "attributeValue" : "240"
  • BBSA_SCRIPT_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_SCRIPT_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as:

    "name":"BBSA_SCRIPT_JOB_STATUS_POLL_INTERVAL_SECONDS"}, "attributeValue" : "60"
  • BBSA_SOFTWARE_JOB_STATUS_POLLS indicates Installing Software (Apache, and so on): The total number of outgoing BSA API calls (from CSM) before the Software Resource constructor operation is timed out in CSM BBSA provider, leading to CSM Software deploy to fail. The value for this parameter is recommended as: 

    "name":BBSA_SOFTWARE_JOB_STATUS_POLLS" },"attributeValue": "240"
  • BBSA_SOFTWARE_JOB_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_SOFTWARE_JOB_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as: 

    "name":BBSA_SOFTWARE_JOB_STATUS_POLL_INTERVAL_SECONDS"}, "attributeValue" : "60"
  • BBSA_VG_AGENT_STATUS_POLLS indicates the status of the BBSA Agent on the VM:  The total number of outgoing BSA API calls (from CSM) to determine the agent status on the provisioned VM before the Virtual-Guest constructor operation times out in CSM, leading to SOI provisioning timeout. The value for this parameter is recommended as: 

    "name" : "BBSA_VG_AGENT_STATUS_POLLS" }, "attributeValue" : "240"
  • BBSA_VG_AGENT_STATUS_POLL_INTERVAL_SECONDS is effective in association with BBSA_VG_AGENT_STATUS_POLLS. It consists of the interval in seconds between any two such outgoing BSA API calls. The value for this parameter is recommended as: 

    "name":"BBSA_VG_AGENT_STATUS_POLL_INTERVAL_SECONDS" }, "attributeValue" : "60"

Note

If the target environment is experiencing high cloning time, increase the poll counts and poll interval values.

BMC Server Automation server tuning

BMC Server Automation is the back-end component of the cloud solution that is used for provisioning VMs and setting them up for usage. BMC Cloud Lifecycle Management uses Virtual Guest Jobs (VGJs) to create VMs and then later runs USP (Update server properties) NSH scripts to fully configure the VMs. It can additionally involve running software deploy jobs.

Use the following set of recommendations to configure and set up the BMC Server Automation application server and job servers for getting a through-put of 250 VMs/hr with a single vCenter.

Assumptions

  • These recommendations assume a through-put of 250VMs/hr using a single vCenter. You can extrapolate any additional throughput requirement, using additional vCenters and job servers.
  • The vCenter is configured so that it can clone and customize a single VM within ~10-12 minutes.
  • The recommendations are for creation of VMs and setting them up; any additional load on BMC Server Automation must be considered separately and accounted for.

Recommendations

  • Use a single Configuration Server with multiple Job Servers to handle various jobs.
  • Configuration Server: Not used as a job server
  • Job Server for VGJs, USP, and Create user jobs (Through-put: 250 jobs/hr - each) - 3 Numbers
    Use the blasadmin -s option.
    • MaxJobs: 75 (per job server)
      For example: 
      blasadmin -s <job_deployment_name> set appserver MaxJobs 75
    • MaxWorkItemThreads: 100 (per job server)
      For example: 
      blasadmin -s <job_deployment_name> set appserver MaxWorkItemThreads 100
  • If BSA Job servers are used for running any other additional types of jobs, then dedicate the previous two job servers for the VGJs, USP, and NSH jobs, by setting affinity rules. You can define affinity rules by using following key words:
    • "VG_Job"
    • "USP_Job"
    • "create_user_job"
  • For all other jobs, configure separate jobs servers depending on the load, using standard BMC Server Automation sizing procedures.
  • Use the blasadmin utility to change the following timeout values for all the job servers, configuration server, and scheduler. Use the blasadmin -a option.
    • BlTargetJobManagerTimeout: 14400 (seconds)
      For example: 
      blasadmin -a set appserver BlTargetJobManagerTimeout 14400
    • MaxJobTimeInSchedulerQ: 300
      For example:
      blasadmin -a set scheduler MaxJobTimeInSchedulerQ 300

BMC Server Automation agent

Increase the number of threads for COServer on the vCenter Server BSA agent by editing the AssetImplConfig.xml file. This file is located at: 
<AgentHomeDirectory>\8.1\RSCD\daal\Implementation\BMC_VMware_VirtualInfrastructureManager_win64\ win64\AssetImplConfig.xml

  • assetImpl_coServerMaxThreadCount = 100
  • assetImpl_coServerFragmentSize = 16384
  • assetImpl_coServerReceiveWaitTime = 30
  • assetImpl_coServerResponseTimeoutTime = 3600
  • assetImpl_cacheTimeout = 3600

Change the following parameters listed in the vmware.properties file. This file is located at:
%install directory%\BMC Software\BladeLogic\RSCD\daal\Implementation\BMC_VMware_VirtualInfrastructureManager_win64\win64.

  • com.bladelogic.vmware.task.timeout= 18000000 (milli seconds)
  • clone-timeout= 18000000 (milli seconds)
  • customization-timeout= 18000000 (milli seconds)
  • customization-timeout-win2k8= 18000000 (milli seconds)

BMC Server Automation agent (Hyper-V)

Increase the number of threads for COServer on the Hyper-V Server BSA agent by editing the AssetImplConfig.xml file. This file is located at:
<AgentHomeDirectory>\8.1\RSCD\daal\Implementation\BMC_VMware_VirtualInfrastructureManager_win64\ win64\AssetImplConfig.xml
Use the BSA Agent values previously listed.

In addition, modify the values of the following parameters in the scvmm.conf file. This file is located at: 
<AgentHomeDirectory>\RSCD\daal\Implementation\BMC_SCVMM_win64\win64

  • TemplateJobRetryCount=100
  • InitialWaitForWin2K8Sec=240

Enterprise-AR and Cloud-AR server tuning

The following table lists the recommended parameters in ar.cfg or ar.conf to configure the Enterprise-AR and Cloud-AR servers.

AR Server tag

Recommended value

Configuration parameter and value

Delay-Recache-Time

300 seconds (5 minutes)

Delay-Recache-Time: 300

Max-Entries-Per-Query returned By GetList

2000

Max-Entries-Per-Query: 2000

Next-ID-Block-Size

100

Next-ID-Block-Size: 100

Server-Side-Table-Chunk-Size

1000

Server-Side-Table-Chunk-Size: 1000

Allow-Unequal-Queries

Disallow

Allow-Unequal-Queries: F

Cache-Mode for application development

Development mode OFF

Cache-Mode: 0

  • Private-RPC-Socket (Fast: 390620) 
  • Private-RPC-Socket (List: 390635)
  • <Value = 3 x # of CPUs>
  • <Value = 5 x # of CPUs>

For example, with a 4 CPU system:

Private-RPC-Socket: 390620 12 12
Private-RPC-Socket: 390635 20 20

The total number of threads should not exceed 8 times the number of CPU cores.

Mid-Tier tuning

The following table lists the recommended values in <Apache_Install_Dir>\Tomcat6.0\conf\server.xml to tune the web infrastructure that hosts the Mid-Tier (the Tomcat servlet).

Tomcat connector parameter

Recommended value

Configuration parameter and value

maxThreads

500

<Connector URIEncoding="UTF-8" acceptCount="250" connectionTimeout="90000" disableUploadTimeout="true" enableLookups="false" maxHttpHeaderSize="8192" maxKeepAliveRequests="-1" maxThreads="500" minSpareThreads="50" port="8080" protocol="HTTP/1.1" redirectPort="8443"/>

BMC ProactiveNet tuning

For BMC ProactiveNet tuning, see:
https://docs.bmc.com/docs/display/public/PN90/Performance+and+scalability+recommendations

Database tuning

Use the following parameters to tune your SQL Server or Oracle database.

MS SQL Server

Based on performance tests conducted in the performance lab, BMC recommends the following Microsoft SQL Server tuning practices:

    • Set PARAMETERIZATION database option to FORCED.
    • Set READ_COMMITTED_SNAPSHOT database option to ON.
  • Set ALLOW_SNAPSHOT_ISOLATION database option to ON.

    Note

    To set ALLOW_SNAPSHOT_ISOLATION and READ_COMMITTED_SNAPSHOT to ON, there must be no active connections to the database except the connection that is executing the ALTER DATABASE statement. When you finish turning on these two database options, multiple connections are supported.

    • Update the statistics regularly by using sp_updatestats stored procedure on SQL Server.
      sp_updatestats updates the query optimization statistics on a table or indexed view. By default, the query optimizer already updates statistics as necessary to improve the query plan. In some cases, you can improve query performance by using UPDATE STATISTICS or the sp_updatestats stored procedure to update statistics more frequently than the default updates.
  • Use the Maximum Degree of Parallelism configuration option (MAXDOP).
    When you configure your SQL Server database server, set the Maximum Degree of Parallelism option to limit the number of processors used in parallel plan execution. By default, this option is set to 0, which uses all available processors. The problem is that one long-running, CPU-intensive SQL statement could monopolize all processors and create long wait times for other users. If your transaction environment shows long-running, CPU-intensive SQL statements, set the max degree of parallelism (MDOP) as follows:

    Number of cores or CPUs on SQL server

    MDOP setting

    1–7

    0 (default)

    8–15

    6

    16 or more

    8

Note

For general information about troubleshooting SQL Server performance, see http://msdn.microsoft.com/en-us/library/dd672789(SQL.100).aspx for MS SQL Server 2008.

Oracle

When tuning Oracle database server, use the following checklist:

  • Make sure that the alert.log is clean.
  • Use an appropriately sized SGA.
  • Set cursor_sharing to FORCE.
  • Set Processes value to 500+ for Enterprise-AR and Cloud-AR database instances.
  • Run gather stats.
  • Use AWR reports to collect diagnostics data. Review the top-timed events for potential bottlenecks.
  • Use an AWR report to monitor the database for the top SQL statement. Make sure that the SQL execution plans are optimal.

Overall system tuning

This section provides guidelines for optimizing the network, client configuration, and overall system.

Performance considerations

Remember the following performance considerations:

  • Create a dedicated environment for the BMC Cloud Lifecycle Management products. Performance is difficult to manage when multiple applications are running in the same environment.
  • Sizing the database server for capacity is important, because it may not be possible to scale the database tier horizontally unless you are using Oracle RAC.
  • Database performance is highly dependent on a fast I/O subsystem. You must have an optimum I/O subsystem.
  • Since the AR System server only caches metadata, it is highly dependent on the fast, reliable, low-latency connection to the database.
  • Use a gigabit connection with close to zero latency between the AR System server and the database. Any additional hop, packet screener, or firewall has a negative impact on performance.
  • On virtualized environments, the vCPU to pCPU ration is workload dependent. For optimized performance, BMC recommends 1:1 vCPU to pCPU ratio.

Note

Resource oversubscription on virtualized hosts, while it does increase virtual machine density, carries with it some risks. After a particular resource is finally exhausted, if that resource happens to be oversubscribed, stability issues can occur and major performance problems can be introduced affecting all of the workloads running on the host server.

Optimizing the network

To optimize the network, BMC recommends the following:

  • Do not install a firewall between the AR System server and the database server, because the firewall has a significant negative impact on the performance.
  • Use a gigabit network with minimal latency between the AR System server and the database server.
  • If there is a load-balancer or firewall between the Mid-Tier and the AR System server, configure it not to sever idle connections from the Mid-Tier. Reestablishing severed connections takes additional time.

Optimum client configuration 

The following lists the optimum client/browser configuration:

  • Dual Core 2+ GHz CPU with 2 MB L2 cache or single core 3+ GHz CPU with 2 MB L2 cache
  • 2-4 GB RAM
  • Firefox 3.6 (or later) or Internet Explorer 8 (or later)
  • Disable all browser add-on toolbars and plug-ins except Adobe Flash.
  • Configure the browser to open pop-ups in a new tab.
  • Configure the browser cache setting to Automatic.

Note

While virus scan is running, all applications will run slower, especially the browser.

Best practices for tuning the overall system

Use the following checklist when tuning your overall system:

  • Survey each area of your system for possible tuning issues before you make any changes.
  • Prioritize all changes by their likely impact on the identified problem.
  • Implement changes one at a time in a prioritized manner.
  • Quantitatively evaluate the impact of every change before you consider whether another change should be made.

Performance tuning checklist

The following table shows the performance tuning checklist for all the BMC Cloud Lifecycle Management components.

Performance tuning checklist
Problem statementCharacterize throughput or response time
  • Before problem was observed
  • After problem was observed
  • Patch changes
  • Configuration changes
  • Workload changes
  • User count changes
 List any system changes that occurred just before the problem was observed
  • Before problem was observed
  • After problem was observed
  • Patch changes
  • Configuration changes
  • Workload changes
  • User count changes
CPU consumption monitoring
  • Collect CPU usage data for each computer in the configuration.
  • Make sure less than 70% of the total CPU capacity is used on each system in the configuration.
  • Record which processes are consuming most of the CPU resources.
  • Record whether I/O wait time is a substantial component of CPU usage. This indicates that the I/O subsystem is not keeping up with demand.
Memory consumption monitoring
  • Make sure no system in the configuration is running out of physical memory and swapping.
  • Be aware of 64-bit and 32-bit limitations on the OS and applications.
  • Track memory growth over time to identify any memory leaks.
BMC Server Automation tuning
  • Allocate 2GB of heap size to the Config server.
  • Depending on the workload, configure additional Job servers.
  • Set appropriate MaxJobs and MaxWorkItem threads for each of these Job servers.
Platform Manager tuning
  • Set appropriate Java Heap Size in wrapper.conf
  • Set appropriate connections and Thread values in cloudservices.json
  • Set appropriate Polling interval and Poll count in providers.json.
Mid-Tier tuning
  • Use precaching to avoid high latency for first-time access to a form.
  • Set resource_check_interval to one day.
  • Set minimum heap size to 1 GB and maximum heap size to 2 GB.
MS SQL Server tuning
  • Set the SQL Server ALTER DATABASE statements.
  • Set the PARAMETERIZATION option to FORCED.
  • Set the SQL Server MDOP option to the appropriate value.
  • Set the AR System server Select-Query-Hint option to NOLOCK.
 
Oracle server tuning
  • Use an appropriately sized SGA.
  • Set cursor_sharing to SIMILAR in Oracle 11g.
  • Use Oracle AWR reports to collect diagnostics.
  • If system is I/O bound or db_file_scattered_read events exist, consider whether high-load SQL statements need tuning or additional indexes should be added
AR System server tuning
  • Set thread counts for FAST and LIST to appropriate values.
  • Set NEXT_ID_BLOCK_SIZE to 100.
  • Use AR System server logging to obtain additional diagnostic information.
  • Set Max Entries Returned by GetList, and disable unqualified searches.
  • Set Server Table Field Chunk Size to reduce the impact of queries with large return sets.
Overall system tuning
  • Survey each area of your system for possible tuning issues before making any changes.
  • Prioritize all changes by their likely impact on the identified problem.
  • Implement changes one at a time in a prioritized manner.
  • Quantitatively evaluate the impact of every change before considering whether another change should be made.
Was this page helpful? Yes No Submitting... Thank you

Comments