[zmsmith]

Here's a pattern I've been guilty of:

1. Set up an alert at a conservative usage to make sure nothing like this can happen

2. See alert and know you have plenty of time to fix the issue

3. Get distracted

4. Disk space disaster

We use AppFirst for our monitoring alerts. One thing they don't support is sending recurring alerts while something is over a threshold. They only send when thresholds are crossed.

Right now we're experimenting with PagerDuty reading the AppFirst alerts and then seeing it as an open issue.

I'd love to know what other people are using.