Enabling resource monitoring for end users

You can enable the ability for users to monitor the health of their services, and take action based on certain conditions. For example, you could enable users to add servers when a service has an overall memory utilization over 90% for 15 minutes, or to send email to a system administrator when CPU utilization exceeds 95% for 10 minutes.

• • Enabling service monitoring
    • Enabling service monitoring as part of a service blueprint
    • Enabling service monitoring as options in the Service Catalog
  • Enabling auto scaling health checks
  • Advanced configuration for service monitoring
    • Monitoring levels
    • Email templates for notification actions
    • To enable hotswap remediation policies
    • Creating custom BMC Atrium Orchestrator workflow actions
  • Enabling debug logging for service monitoring
  • Related topics 

See the following BMC Communities videos to learn about scaling services.

 BMC CLM vSphere Horizontal Scale Demo (4:21): https://youtu.be/yiWVCyk1Tss

Service Elasticity (6:37): https://youtu.be/NcQouCo-1ZI

See also the following BMC Communities videos series about monitoring and remediation:

Health Monitoring and Remediation of Servers and Services - Part 1 (5:20): https://youtu.be/kyA4bbNn8fs

Health Monitoring and Remediation of Servers and Services - Part 2 (5:49): https://youtu.be/67pHNzk0XjY

Health Monitoring and Remediation of Servers and Services - Part 3 (6:27): https://youtu.be/xv7Gv3kYCBA

Health Monitoring and Remediation of Servers and Services - Part 4 (8:23): https://youtu.be/e2ZZ9igO-hw

Health Monitoring and Remediation of Servers and Services - Part 5 (5:53): https://youtu.be/qAjUvS3o6eY

Enabling service monitoring

By default, service monitoring is not enabled for any services. You do not need to make any configuration changes to the default settings present when you install BMC Cloud Lifecycle Management. You only need to enable service monitoring, which you can do in two ways:

• As a native part of a service blueprint—All services provisioned from the service blueprint have service monitoring enabled.
• As options in the Service Catalog—You can allow users to enable service monitoring as a part of the initial service request, a post-deployment action, or both.

Enabling service monitoring as part of a service blueprint

1. In the Service Designer workspace, select a service blueprint from the Blueprint Library.
   The service blueprint appears in view-only mode.
2. Click Check Out.
   A local copy of the service blueprint is created and is available to edit.
3. Click Definition > Properties to open the Definition Details dialog box.
   

4. On the Properties tab, enter information for the following fields:

   Field	Action
   Enable Monitoring	Select the Enable Monitoring check box to enable performance monitoring of the resources provisioned by the service blueprint. Selecting this check box also allows users to view, create, and edit charts for their services through the My Cloud Services console. See Managing-servers-in-a-service for more information. By default, this check box is cleared. Note: This is not applicable for Azure Resource Manager even if the check box is selected.
   Monitoring Level	Select a Monitoring Level. By default, the Gold level is selected.
   Enable Monitoring Policy	Select the Enable Monitoring Policy check box to allow users to create monitoring rules in the My Services console. If you select the Enable Monitoring check box, you can also enable policies for that service. If policies are enabled, end users can create rules that perform actions based on thresholds being breached. For example, a user might create a rule to notify an administrator when the memory utilization of a service exceeds 95% for 15 minutes. Rules also enable users to take remedial action, such as adding capacity when thresholds are breached. See Managing-cloud-resources for more information. You can control the amount of resources that can be added through rules by adjusting quota. See Setting-and-managing-quota for more information. By default this check box is cleared.

   After monitoring is enabled, the My Services console reflects the AWS CloudWatch metrics in the service health charts. 

5. Select File > Save and Check In.
6. Click OK.
   The blueprint is removed from the My Checked Out Blueprints list, and is added to the Blueprint Library.

See Creating-copying-or-editing-a-service-blueprint for more information about service blueprints.

Enabling service monitoring as options in the Service Catalog

You can define Blueprint Configuration options in the Service Catalog that allow users to request service monitoring as part of a service request, after a service is provisioned, or both. Using options prevents service monitoring from being enabled by default, but still allows monitoring where users deem it necessary. You might also want to price monitoring as an additional cost of the service, and present that as an option for the service. The Blueprint Configuration option type is Monitoring. When defining this option you specify the same settings as you would using the options native to the service blueprint, namely Enable Monitoring, Enable Monitoring Policy, and Monitoring Level.

Similarly, you can create an option that allows users to disable service monitoring.

See Configuring-end-user-Option-Choices-in-service-blueprints and Option-Choices-available-when-extending-service-blueprints for more information.

Enabling auto scaling health checks

Implementing auto scaling enables you to scale your services in a virtual private cloud (VPC) by increasing the resources on individual servers (scaling up or down) or by adding new servers (scaling out or in) automatically, according to conditions you define through the BMC Cloud Lifecycle Management Service Monitoring feature. With auto scaling, you can ensure that the resources of the instances in use increase seamlessly during demand spikes to maintain performance, and decrease automatically during demand lulls to minimize costs. Auto scaling is useful for applications that experience variability in usage.

The following auto scaling actions are available for Amazon Web Services and VMWare:

• Add memory with server restart

• Remove memory with server restart

• Add CPUs with server restart

• Remove CPUs with server restart

• Add servers

• Remove servers

To enable auto scaling, you must first enable monitoring on your services. Based on frequency (interval between the data retrieval of consecutive metrics such as CPU, memory, and so on), BMC Cloud Lifecycle Management supports the following levels of monitoring:

• Gold level – Usually maps to a refresh interval of 1 minute.
• Silver level – Usually maps to a refresh interval of 5 minutes.
• Bronze level – Usually maps to a refresh interval of 10 minutes.

You can change the monitoring level to the supported Refresh Frequency mapping by changing access attribute value in the cloudservices.json file as described in Modifying monitoring levels.

End users (owners of the respective services, who define the auto scaling rules and actions) can define the following parameters:

• CPU and memory upper limits and delta to be added or removed
• Server count to be added or removed as well as the upper and lower bounds (add or remove servers)

Advanced configuration for service monitoring

If you want to expand service monitoring capabilities, complete these activities:

• Add new monitoring levels.
• Modify the default email notification template.
• Determine what remediation actions you want enabled, and enable those actions.
• Determine whether you want to make BMC Atrium Orchestrator workflow available as custom actions in service monitoring rules.

Monitoring levels

Monitoring levels define how often BMC Cloud Lifecycle Management assesses and manages the operational state of service offering instances. BMC Cloud Lifecycle Management supports multiple monitoring levels, reflecting the need to manage different service categories with the right level of operational assessment. For example, you might want service levels for categories such as business critical, development, test, or customer service-level agreements such as Gold, Silver & Bronze. When you enable service monitoring for a service blueprint, you also select a monitoring level.

Default monitoring levels

By default, BMC Cloud Lifecycle Management includes the following monitoring levels:

• Gold—1 minute frequency
• Silver—5 minute frequency
• Bronze—10 minute frequency

If you have enabled service monitoring, users can create and edit charts to retrieve data at a polling interval specified by those users. Note that the user-defined polling interval in a chart can be different than the interval specified by a monitoring level. The polling intervals available to users on charts are multiples of the administrator-defined collection interval specified by a monitoring level.

Modifying monitoring levels

BMC Cloud Lifecycle Management defines monitoring levels in the PlatformManager/Configuration/cloudservices.json file. You can add, modify, or delete monitoring levels by editing the SOIBucketsCfg attribute of the OpsManager service in that json file. The attributeValue line defines monitoring levels. The following example shows the portions of the cloudservices.json file relevant to service monitoring levels:

To modify monitoring levels

1. Stop the BMC Cloud Lifecycle Management Platform Manager service:
   • (Windows) In the Windows Services window, select the BMC CSM service, and click Stop the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/bmccsm stop
2. Open the PlatformManager/Configuration/cloudservices.json file for editing.
3. In the cloudservices.json file, locate the SOIBucketsCfg attribute of the OpsManager service.

4. In the attributeValue line for the SOIBucketsCfg attribute, add or modify monitoring levels.
   Each level is in the format <bucket-name>,<tag-group>,<tag>,<SummarizationPeriod>, with semicolons separating the different levels. The following table explains each piece of a level's definition:

   Variable	Definition
   <bucket-name>	An internal name for the monitoring level. This name does not appear in the BMC Cloud Lifecycle Management UI.
   <tag-group>	Monitoring levels are treated as tags in a tag group. The tag group must be MonitoringLevel.
   <tag>	A display name for the monitoring level, which can be the same as the <bucket-name>. This name appears in the BMC Cloud Lifecycle Management UI.
   <SummarizationPeriod>	The data-collection interval, in seconds. The minimum number is 60. The interval must be a whole-minute interval (divisible by 60).

   For example, if you want to add a new level called Coal with a data-collection interval 30 minutes (1800 seconds) long, you would edit the attributeValue line to appear as in the following example:

   "attributeValue" : "Gold,MonitoringLevel,Gold,60;Silver,MonitoringLevel,Silver,300;Bronze,MonitoringLevel,Bronze,600;Coal,MonitoringLevel,Coal,1800",
5. Save and close the cloudservices.json file.
6. Start the BMC Cloud Lifecycle Management Platform Manager service:
   
   • (Windows) In the Windows Services window, select the BMC CSM service, and click Start the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/bmccsm start

Email templates for notification actions

BMC Cloud Lifecycle Management includes email templates that are used for notification actions. The templates are Velocity Template Language (VTL) files located in the Platform_Manager/configuration/email-templates folder. In this folder are the following items:

• UILabels folder—Contains 2 properties files (in English and simplified Chinese) that define how UI labels appear in notification emails.
• default_template.vm file—Do not modify this file. It is reserved as a default template from which you can create other language-specific templates.
• EmailTemplate_en.vm file—The English-language version of the template.
• EmailTemplate_zh.vm file—The simplified Chinese-language version of the template.

You can modify the language-specific templates to suit the needs of your environment and your cloud users.

To modify an email notification template

1. Stop the BMC Cloud Lifecycle Management Platform Manager service:
   • (Windows) In the Windows Services window, select the BMC CSM service, and click Stop the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/bmccsm stop
2. In a Velocity editor, open the Platform_Manager/configuration/email-templates/EmailTemplate language-specific file you want to modify.

3. Edit the file and save your changes.
   For example, you might add the following line to convey the duration until the action can be triggered again, as determined by the rule:

   <tr><td class="lable">&nbsp; &nbsp; Another violation of this condition will be ignored for&nbsp;</td>
   <td class="value">${refreactoryperiod}&nbsp;mins</td></tr>
   
4. Start the BMC Cloud Lifecycle Management Platform Manager service:
   
   • (Windows) In the Windows Services window, select the BMC CSM service, and click Start the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/bmccsm start

To enable hotswap remediation policies

BMC Cloud Lifecycle Management supports the ability for users to create service monitoring rules that add CPUs and memory without restarting the server, commonly called a hotswap. These options are disabled by default. If you have enabled support for the hotswap of CPUs and memory in your VMware environment, you can enable those service monitoring actions in BMC Cloud Lifecycle Management.

1. Stop the BMC Remedy Action Request System Server service:
   
   • (Windows) In the Windows Services window, select the BMC Remedy Action Request System Server service, and click Stop the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/arserverd stop
2. Stop the BMC Cloud Lifecycle Management Platform Manager service:
   • (Windows) In the Windows Services window, select the BMC CSM service, and click Stop the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/bmccsm stop
3. Open the PlatformManager/Configuration/actionCatalog/cloud_actionCatalog.json file for editing.

4. In the catalogGroups section, add the following lines:

   {
          "name": "VerticalAutoScaleWithHotSwap",
          "label": "AutoScaleResourceSet without restarting the servers",
          "applicableClass": "ResourceSet",
          "fillFields": "*,compute.*,compute.server,compute.localDisks",
          "actionMetaData": [
           {
              "name": "AutoScaleResourceSetHotSwapActionMetaData",
              "description": "ActionMetaData for VerticalAutoScaleWithHotSwap",
              "actionMetaDataNameValuePair" : [
                   {
                      "name": "compute.server.vendor",
                      "value": "VMWare"
                   }]
           }],
          "catalogGroupInputParams": [{
              "label": "targetObjectId"
           }],
          "catalogEntries": [
           {
              "name": "ScaleUpMemoryHotSwap",
              "label": "Scale Up Memory without restarting the server",
              "actionType": "APIInvocation",
              "API": "ResourceSet.modifyServer",
              "APIType": "Instance",
              "HTTPType": "POST",
              "catalogEntryInputParams": [{
                  "label": "increment",
                  "defaultValue": "true",
                  "valueDataType" : "BOOLEAN"
               },
               {
                  "label": "doHotSwap",
                  "defaultValue": "true",
                  "valueDataType" : "BOOLEAN"
               },
               {
                  "label": "changeMemoryBy"
               },
               {
                  "label": "maxMemoryAllowed"
               }]
           },
           {
              "name": "ScaleUpCPUHotSwap",
              "label": "Scale Up CPU without restarting the server",
              "actionType": "APIInvocation",
              "API": "ResourceSet.modifyServer",
              "APIType": "Instance",
              "HTTPType": "POST",
              "catalogEntryInputParams": [{
                  "label": "increment",
                  "defaultValue": "true",
                  "valueDataType" : "BOOLEAN"
               },
               {
                  "label": "doHotSwap",
                  "defaultValue": "true",
                  "valueDataType" : "BOOLEAN"
               },
               {
                  "label": "changeCPUCountBy"
               },
               {
                  "label": "maxCPUAllowed"
               }]
           }]
         }
5. Open the PlatformManager/Configuration/actionCatalog/Reference_ActionCatalog.json file for editing.

6. In the catalogGroups section, add the following lines:

   {
              "name": "VerticalAutoScaleWithHotSwap",
              "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap@",
              "description": "Handles the Scale Actions for Group of Servers (ResourceSet) without restarting them",
              "applicableClass": "ResourceSet",
              "targetContext": "",
              "catalogGroupInputParams": [
                   {
                      "name": "targetObjectId",
                      "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.inputParam.targetObjectId@",
                      "valueDataType": "String",
                      "isOptional": false,
                      "isEndUserInput": false,
                      "sequence": "1"
                   }
               ],
              "catalogEntries": [
                   {
                      "name": "ScaleUpMemoryHotSwap",
                      "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.actionEntry.ScaleUpMemoryHotSwap@",
                      "description": "Add more Memory for all Servers in the server group without restarting the servers",
                      "actionType": "Remediation",
                      "catalogEntryInputParams": [
                           {
                              "name": "changeMemoryBy",
                              "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.actionEntry.ScaleUpMemoryHotSwap.inputParam.changeMemoryBy@",
                              "description": "Scale Up memory for each Server in the server group by specified MB without restarting the server",
                              "valueDataType": "Long",
                              "isOptional": false,
                              "isEndUserInput": true,
                              "sequence": "1"
                           },
       {
                              "name": "limitMemoryChange",
                              "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.actionEntry.ScaleUpMemoryHotSwap.inputParam.limitMemoryChange@",
                              "description": "Memory may scale up to this amount without restarting the server",
                              "valueDataType": "Long",
                              "isOptional": false,
                              "isEndUserInput": true,
                               "sequence": "2"
                           }
                       ]
                   },
                   {
                      "name": "ScaleUpCPUHotSwap",
                      "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.actionEntry.ScaleUpCPUHotSwap@",
                      "description": "Add more CPUs for all Servers in the server group without restarting the servers",
                      "actionType": "Remediation",
                      "catalogEntryInputParams": [
                           {
                              "name": "changeCPUCountBy",
                              "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.actionEntry.ScaleUpCPUHotSwap.inputParam.changeCPUCountBy@",
                              "description": "Scale Up CPU for each Server in the server group by specified number without restarting the server",
                              "valueDataType": "Integer",
                              "isOptional": false,
                              "isEndUserInput": true,
                               "sequence": "1"
                           },
       {
                              "name": "limitCPUCount",
                              "label": "@action.catalogEntry.VerticalAutoScaleServerGroupWithHotSwap.actionEntry.ScaleUpCPUHotSwap.inputParam.limitCPUCount@",
                              "description": "CPU may scale up to this value without restarting the server",
                              "valueDataType": "Integer",
                              "isOptional": false,
                              "isEndUserInput": true,
                              "sequence": "2"
                           }
                       ]
              }]
      }
7. Delete the following 2 cache files in the PlatformManager/cachefolder:
   • NotificationProviderInstance.dat
   • CloudActionProvider.dat
8. Start the BMC Remedy Action Request System Server service:
   
   • (Windows) In the Windows Services window, select the BMC Remedy Action Request System Server service, and click Start the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/arserverd start
9. Start the BMC Cloud Lifecycle Management Platform Manager service:
   
   • (Windows) In the Windows Services window, select the BMC CSM service, and click Start the service.
   • (Linux) From a command prompt, type the following command:
     /etc/init.d/bmccsm start

Creating custom BMC Atrium Orchestrator workflow actions

By default, if you have enabled monitoring policies users can create rules that initiate the following actions:

• Add or remove memory, with or without a server restart
• Add or remove CPUs, with or without a server restart
• Add or remove servers

Additionally, you can use the BMC Cloud Lifecycle Management API to enable BMC Atrium Orchestrator workflow to be triggered by a rule. For example, you might want to create an action that restarts a service on a VM. Any custom workflow available to be used in a monitoring rule appears to users in the Custom category below the default actions.

To create a custom BMC Atrium Orchestrator workflow action

1. On the BMC Cloud Lifecycle Management Platform Manager computer, open the providers.json file for editing.
2. Ensure that the Callout Provider can communicate with the BMC Atrium Orchestrator computer by setting the following attributes in the providers.jsonfile:
   • AO hostname
   • Port
   • Grid name
   • Username
   • Password
3. Save and close the providers.json file.

4. Using a REST client, log in to the Platform Manager, using the following information as part of your request:

   HTTP Method	POST
   URL Pattern	http://PlatformManagerURL:PlatformManagerPortNumber/csm/login
   Request Body example	{ "username" : "<cloudadmin-username>", "password": "<cloudadmin-password>" }

   Note the Authentication-Token in the response header, and use it in the header of subsequent requests.

5. Using a REST client, make your BMC Atrioum Orchestrator workflows available as BMC Cloud Lifecycle Management actions. Use the following information as part of your request, where value is a comma-separated list of BMC Atrium Orchestrator modules and workflows. If you specify a module, all workflows in that module are converted to actions.

   HTTP Method	POST
   URL Pattern	http://PlatformManagerURL:PlatformManagerPortNumber/csm/ActionCatalog/refresh
   Request Body example	{   "operationParams":[      {         "name":"criteria",         "type":"java.lang.String",         "multiplicity":"1",         "value": "<modulesAndWorkflows>"      }    ],   "timeout" : -1 }

   The PlatformManager/Configuration/actionCatalogs/delta directory is created. This directory is a temporary directory that allows you to make any necessary edits to the actions before introducing them into the main BMC Cloud Lifecycle Management processes and interfaces. The delta directory contains the following files:

   • CLMCustomUILabels.properties
   • Custom_Action_Catalog.json
   • Reference_ActionCatalog.json

   Note

   Running this API again overwrites the contents of the delta directory. If you need to run this API call again, be sure that you list all needed module and workflows in your request.

6. (optional) If you need to add parameters to your custom actions, open the Reference_ActionCatalog.json file for editing and enter parameters where needed:

   1. For example, if you want to encrypt a password, locate the password parameter in the json file and enter isEncrypted as true:

      "name" : "password",
      "isEncrypted" : true
             "sequence" : "3",
             "label" : "@action.catalogEntry.Custom.<Name>.Compensate.password.label@",
             "valueDataType" : "String",
             "isEndUserInput" : true

   2. If your BMC Atrium Orchestrator workflow includes purely contextual information as part of the action (such as the name of the service that triggered the action), to prevent that information from appearing in the BMC Cloud Lifecycle Management interface as options to the end user locate the AdditionalInfo parameter in the json file and enter isEndUserInput as false:

      "name" : "AdditionalInfo",
             "sequence" : "1",
      "isEndUserInput" : false
             "isOptional" : true,
             "label" : "@action.catalogEntry.Custom.<Name>.Failure.AdditionalInfo.label@",
             "valueDataType" : "String"
   3. If you want to reorder the sequence in which parameters appear in the BMC Cloud Lifecycle Management interface, adjust the sequence value accordingly.
   4. Save and close the Reference_ActionCatalog.json file

7. Using a REST client, merge the actions from your delta directory into the main BMC Cloud Lifecycle Management processes and interfaces:

   HTTP Method	POST
   URL Pattern	http://PlatformManagerURL:PlatformManagerPortNumber/csm/ReferenceActionCatalog/refreshCatalogs

8. (optional) If you need to localize your options for non-English languages, complete the following steps:
   1. Make a copy of the PlatformManager/Configuration/actionCatalogs/delta/CLMCustomUILabels.properties file for each language you need, changing the file name to include an underscore and shorthand for that language. For example, if you need to display your actions in Italian, you would create a file named  CLMCustomUILabels_it.properties. The shorthand values can be any of the following:
      • Brazilian Portuguese—pt-br
      • English—en
      • French—fr
      • German—de
      • Italian—it
      • Japanese—ja
      • Korean—ko
      • Russian—ru
      • Simplified Chinese—zh-cn
      • Spanish—es
   2. Change the name of original CLMCustomUILabels.properties file to CLMCustomUILabels_en.properties to differentiate it as the English-language file.
   3. Change the string values in your properties files to use the appropriate languages.
9. On a computer running JDK 1.6 or higher, run the following command to jar the language files on the Platform Manager computer. Do this even if you are using only the default, English-language CLMCustomUILabels.properties file.
   jar cf ServiceHealthCustomUILabels.jar *.properties
10. On the Platform Manager, log in as a BMC Remedy Action Request System (AR System) administrator and open the Data Visualization System Files form.
    In a browser, use the following URL format: http://serverName:portNumber/arsys/forms/serverName/Data+Visualization+System+Files/
11. Open a new request and enter information as specified in the following image:
    
12. Attach the ServiceHealthCustomUILabels.jar file to your request, and then save the request.
13. Restart the BMC Remedy AR System Mid Tier.

Enabling debug logging for service monitoring

To enable debug logging for service monitoring, set the following logging categories to the debug level, as described in Working-with-category-debug-logging.

• RULESMANAGER_ERPROVIDER
• NOTIFICATION_PROVIDER
• CUSTOM_ACTION_PROVIDER
• OPINVOCATION_PROVIDER
• OPS_COMPOSITE_PROVIDER
• OPS_DATA_ANALYSIS_PROVIDER
• AUTOPILOT_PROVIDER
• OPS_MANAGER
• SVCDATAANALYSISPROVIDER
• OPS_VMWARE_PROVIDER
• RM

Related topics 

Managing-servers-in-a-service
Managing-cloud-resources 
Creating-copying-or-editing-a-service-blueprint