Twenty One Prophecies
There is a fable the analytics industry keeps retelling with fresh data. A shepherd boy cried wolf for sport until the village stopped listening, and the wolf arrived. Replace the boy with an alerting system, the wolves with weekly thresholds, and the village with an operations team, and the story runs every Monday.
21 alerts are waiting for review in a Monday inbox. 14 concern Sunday evening, when traffic always dips, because Sunday is Sunday. 5 belong to a metric nobody remembers approving, firing on schedule since someone abandoned it a year ago. The last 2 warn of the same revenue threshold in two different fonts. Not one of them required a person to do anything, and by the third unread week, nobody did.
Then the wolf arrives. A Tuesday release breaks the checkout flow, and revenue begins bleeding hourly. The alerting system fires on time. But the message enters an inbox that has cried wolf for six consecutive weeks, and it sits between two Sunday false alarms before anyone opens it. By then the crisis has peaked, and a system built to be first is reliably last.
Here is the uncomfortable truth of that scene. Alerts are not a notification system. They are a trust system. Every false positive erodes confidence exactly the way a false prophecy does, and every missed signal erodes revenue while the team learns to ignore the messenger.
The Line Drawn Through Sunday
The default instrument of alerting is the threshold. Draw a fixed line, notify whoever crosses it. Simple to build, and fatally simple to break.
Revenue below 10,000 sounds like prudence until the first Sunday evening falls beneath it, because Sunday evenings always do. The threshold knows the number. It does not know the day, the season, or the promotion calendar. The analyst is then handed two dials, and both fail. Loosen the line, and the alert stays silent through the genuine collapse. Tighten it and Monday morning drowns in Sunday’s false alarms. Either way, the baseline is wrong, because a seasonal business cannot be guarded by a single number.
The Baseline That Learns
Anomaly detection replaces the line with an expected range. Instead of asking whether revenue has crossed 10,000, it asks a harder question. What does revenue usually look like for this day, at this hour, in this season, and how much noise does that normal tolerate?
The model learns from history, 35 days of daily data matched against the same window a year before, so that last year’s autumn teaches this year’s autumn. It then draws a moving band of expectation and fires only when reality escapes the 95% confidence interval around it. Sunday dips stop being emergencies, because Sunday’s Sunday was always going to dip. A Tuesday collapse in a quiet September, on the other hand, fires immediately, because no Sunday ever threatened a Tuesday like that.
Two honest footnotes belong in this promise. Anomaly-based alerting requires the Prime or Ultimate package, so teams on Foundation or Select should spend their vigilance on carefully placed thresholds instead. And no learning removes the need for judgement about which metrics deserve watching at all.
The Alert Is the Question, Not the Answer
An alert says look here. It never says why. Its value therefore depends entirely on what happens next, and the rule for alerting is ruthless. Every alert must have a clear, documented action, or it should not exist at all.
When the action is investigation, the chain is already built. From an anomalous cell in Analysis Workspace, one click opens contribution analysis, which scans every dimension in the suite and ranks the items most responsible for the deviation in seconds. The hunt that used to consume an afternoon, one analyst drilling through browsers, campaigns and regions on instinct, returns machine-ranked. On a sudden spike of checkout errors, four-fifths of the anomaly traced to a single browser version in a single market, carrying a single regional promo code. That precision took a click, not hours.
So the chain runs short. Anomaly found, alert delivered, contribution opened, cause named, engineer briefed. When the chain is that short, an alert is no longer noise. It is the start of a decision, and the decision is where the value lives.
The Alert Audit
The test costs nothing but an afternoon. Take every alert the team currently runs and ask one question. What action does this alert produce?
Write the answer in a sentence. Brief the engineer, suppress the campaign, open the investigation. If the sentence writes itself, the alert holds a job. If the sentence requires stretching, the alert is furniture that generates email, and it should be retired before it retires the team’s attention.
Count while at it. Last month’s alerts beside last month’s actions give the trust ratio. A team that acts on most of its alerts runs a quiet inbox. A team that acts on one in ten has stopped reading, and stopped reading is how the wolf walks in.
Want to Know More?
If you want to know more, check out Adobe Analytics: A Champion’s Handbook. The AI and machine learning chapters go deeper than this post can, with the confidence intervals behind anomaly detection, the path from anomalous cell to contribution table, and the alert governance that keeps every notification worthy of trust. And if your battleground lies elsewhere, the handbook walks the whole journey, from data collection and workspace analysis to APIs, governance and the road to Champion.
Leave a Reply