Analysis
Alert
Also called threshold alert.
Alert is a message sent when a metric crosses a set line or breaks its usual pattern. It exists to wake a person up, not to report.
How it is measured
An alert has four parts: the metric, the condition, the evaluation window, and the recipient. A threshold alert fires when a value passes a fixed line for a set time. A pattern alert fires when the value leaves a band built from past data.
Measure the alert itself by how often it fires and how often anyone acts. Over a month, count the pages sent, the ones that were real, and the time between the event and the first human response.
Worked example
An events-ticketing site sets an alert on checkout completions: fewer than 8 in any 30-minute window during sales hours. At 2:10 on a Thursday afternoon it reads 3, then 1, then 0.
The on-call developer finds a payment script returning a 500 error after a deploy and rolls it back by 2:40. Without the alert the gap would have been found by the morning report, hours of ticket sales later.
How it differs
An alert interrupts a person about a condition now. An anomaly detection model is the method that decides what counts as unusual. An alert can use a plain threshold with no model at all.
Common errors
Setting thresholds so tight that the channel gets muted. Alerting on every metric. Sending the page to a shared inbox no one reads. Setting no recovery message. Never reviewing which alerts led to action.
In practice
Pick the three numbers whose silence would cost you money, such as orders, signups, or uptime. Give each one a recipient, a window, and a written first step. Retire any alert that fired five times with no action.