Threshold management using operator console
Absolute thresholds
Absolute thresholds are simple and static sets of thresholds that represent absolute values above or below which an event is generated. In general, you specify absolute thresholds attributes that have commonly-accepted values beyond which performance is known to degrade. Absolute thresholds are better suited for attributes that change status.
For example, when the total CPU utilization of a Solaris system exceeds 80%, performance issues can occur. In this case, you would specify an absolute threshold of 80% for this attribute.
Absolute thresholds' event-triggering behaviors are easier to grasp because of the requirement that all performance data points must exceed the trigger condition for a specified duration before an event gets created. The absence of the trigger condition for the same duration will close the event.
Having the baseline in the trigger condition allows a threshold to generate events based on learned behavior.
The following figure illustrates a typical trigger condition for an absolute threshold.
Absolute thresholds with outside baseline enabled
Absolute thresholds with outside baseline are thresholds which combine both absolute thresholds (static threshold value) and dynamic thresholds (which use the baseline) as the threshold. The user can still set the threshold value with but as additional criteria for alarm generation.
Advantage:
- Reduce the number of false alarms when compared with using static threshold.
- Absolute thresholds with outside baseline allows more control than signature thresholds.
Signature thresholds
Where Absolute thresholds tend to fail to create events are scenarios where the performance data take on a more transient behavior, rendering the trigger condition less obvious. Signature thresholds can be utilized here because it can make a judgment when enough data points are needed to trigger an event, as illustrated below.
Due to the non-deterministic nature of the performance data, it might take longer than anticipated for a Signature threshold to generate an event.
Signature threshold for performance metrics that have no set concept of value is completely dependent on the attribute and the instances (transaction response time, Ping Response time, etc) being monitored.
Internal events and their life cycles
Events that are generated by the TrueSight Infrastructure Management Server (by the three native thresholds) are called Internal events. They are marked with icons having a double wrench. The key characteristics of Internal events are:
- Thresholds of the same type (Absolute, Signature, or Abnormality) for a metric of an instance operate on the same event, A.
- Event A has a long life span, maintaining the same mc_ueid.
- Besides creations and closures, state changes such as severity demotion/promotion are also legitimate changes for event A.
When to use thresholds
KPIs, baselines, and thresholds
- Baseline generation requirements
- Which Baseline to Use
- Seasonality baselines
- How data affect thresholds
- Advanced Signature threshold configuration
- Prediction
- How to Persist Thresholds
Baseline generation requirements
Key Performance Indicators (KPIs) are essential metrics for monitoring an infrastructure. They have a direct impact on whether or not baseline computation takes place for corresponding metrics. The following figures show how KPIs may affect baseline generation where the checked boxes indicate that baseline generation gets carried out for those combinations.
In order to function correctly, Abnormality and Signature thresholds require baseline data. Due to this requirement, you may face support issues as certain thresholds would not work. In such cases, ensure that you verify that the baseline is being generated for the metrics in question.
Which Baseline to Use
The TrueSight Infrastructure Management Server automatically computes three different types of baselines (Hourly, Daily, and Weekly) to be used by the thresholds. In most cases, when defining thresholds, it is adequate to use Auto Baseline, where the TrueSight Infrastructure Management Server determines the best type of baseline to use for any given metric.
However, if it is known that certain metrics have clear, repeatable hourly patterns (for example, 10 AM on Tuesday behaves in the same way as 10 AM on Wednesday), then you can select Hourly Baseline as the baseline type to use for those corresponding thresholds. Similarly, Daily and Weekly baselines can be used by thresholds if you know that their metrics behave accordingly.
Seasonality baselines
This feature is useful for infrastructures that have recurring periods where (part of) the infrastructure behaves very differently and that they do not want these behaviors to be factored into the normal baselines. Example: Cases where you have a major back-up on the last Friday of every month, financial number crunching at the end of every quarter, and so on.
In order for Seasonality baselines to work properly, the Infrastructure Management administrator must ensure that the baseline retention period properly reflects the special recurring period. For example, if the recurring period is twelve-month long, the baseline retention period has to be just as long.
How data affect thresholds
Thresholds are data-driven – the more available data points, the sooner thresholds can generate some events, especially those that make use of baselines. However, frequent polling intervals will increase the TrueSight Infrastructure Management Server's system load.
Advanced Signature threshold configuration
When creating a Signature threshold, it is desirable to fine-tune the behavior of the threshold. As shown in the following image, there are four additional fields that become visible when you select the advanced view when creating Signature thresholds.
The field descriptions are:
Field Name | Description and usage |
|---|---|
Minimum Sampling Window | The minimum span of time, as marked by collected data points, required in order for the Signature threshold engine to initiate evaluation. |
Threshold | Specified if you do not want an algorithm to trigger on trivial conditions. For example, if the baseline is low (around 3% - 5%), specify a high threshold value so that Signature thresholds will only trigger if data values are higher than 80% and surpassing baseline. |
Deviation | Use this section to expand the baseline value range; typically, to reduce the sensitivity of the Signature threshold. |
Prediction
The Prediction feature gives early warnings of certain exceptional situations and is used to issue warnings if there is an aggressive trend towards the threshold. While it can be used in a wide number of scenarios, it is most effective in capacity-type scenarios, especially for those metrics which exhibit clear hourly patterns.
Some of the points to remember when using this feature include:
- Only applicable to Absolute thresholds
- Baseline is required
- Will need to increase polling interval - easier and more timely to establish trends
- Will use up more run-time resources. Have to examine load if large number of thresholds enables Prediction
How to Persist Thresholds
Use the pw threshold checkpoint command to save states of threshold while customizing deployment.