Setting Alert thresholds

You must choose your threshold carefully. If you set a threshold that is too high, the action program is not executed in a timely fashion. Likewise, if the threshold is too low, the action program is executed too often, creating false positives.

A threshold of 3 counts per 60 seconds is often the exact setting required to generate meaningful alerts, especially when a message occurs relatively infrequently, or a message is sporadically received

This value of three has a solid mathematical basis, as follows:

• If the standard deviation of a data set over a relatively small interval is less than one, and the average of that data set is close to zero, then the probability distribution of the data is best given by the Gauss Error Function (also denoted as erf(x)).
• The probability of three messages occurring over that interval is, 1 - erf(3 / sqrt(2)), that evaluates to approximately 0.5% of all the sampled time intervals.

Therefore, for this type of typical data and with a sample interval of 60 seconds, the typical alert is triggered approximately once every three hours or less.

If both the average value and standard deviation for the sampled data are greater than one, then a regular Gaussian normal distribution provides a more appropriate estimate of an alert occurs. In this case, you can select three standard deviations away from the average for a meaningful alert, that again evaluates to approximately 0.5% of all sampled time intervals.