Setting Alert thresholds

As previously discussed, the alert threshold is fairly important. If you select a threshold that is too high, then the action program is not executed in a timely fashion. Likewise, if the threshold is too low, then the action program is executed too often, creating false positives.

In practice, it is not as difficult to set alert thresholds as one might think it is. As suggested before, a threshold of 3 counts per 60 seconds is often the exact setting required to generate meaningful alerts, especially when a message occurs relatively infrequently, or a message is sporadically received

This value of 3 actually has a solid mathematical basis as follows:

• If the standard deviation of a data set over a relatively small interval is less than 1, and the average of that data set is close to zero, then the probability distribution of the data is best given by the Gauss Error Function (also denoted erf(x)).
• The probability of 3 messages occurring over that interval is, 1 - erf(3 / sqrt(2)), that evaluates to approximately 0.5% of all the sampled time intervals.

Therefore, for this type of typical data and with a sample interval of 60 seconds, the typical alert is triggered approximately once every three hours or less.

If both the average value and standard deviation for the sampled data are greater than 1, then a regular Gaussian normal distribution provides a more appropriate estimate of an alert occurrs. In this case, you can select three standard deviations away from the average for a meaningful alert, that again evaluates to approximately 0.5% of all sampled time intervals.