It worked. For 92 of them.
You know me; I hate being blind to what's going on with the server. After years of development, every new failure weighs a little bit more. Not because the failures get worse. Because you start to recognize them. You've seen this one before, you fixed it before, and here it is again wearing a slightly different shirt.
This one I hadn't seen before, and it bothered me more than most.
We have a job that sends notification emails to customers. Around a hundred recipients at a time. It runs, exits 0, pings its heartbeat, and the CleverQA dashboard stays green. Done.
Except it sent 92 emails, not 100.
We found out the way nobody wants to find out: support tickets. Five of them, over a couple of days, all saying some version of "I never got the email." Five tickets for eight missing emails, which means at least three people never bothered to tell us. I still think about those three. They just assumed we didn't care enough to write.
The job hadn't crashed. It hadn't hung. From outside it looked exactly like every other night. There was nothing to alert on, because by every signal we were collecting, nothing was wrong. That's the part that got under my skin. I wasn't blind because I had no monitoring. I was blind with monitoring.
"It ran" is not the same as "it worked"
This took me longer to accept than it should have. Batch jobs almost never fail all at once. A total failure is actually the easy case: the process dies, the exit code is non-zero, the heartbeat goes quiet, somebody gets paged. Loud problems get fixed, because loud problems get noticed.
What batch jobs actually do is fail a little. A loop over a hundred records where eight hit a provider that says no. A sync where one tenant's token expired and the other forty are fine. An export where a handful of rows have a null in a column that was never supposed to be null. The loop catches the exception, writes a log line, moves on to the next record, and finishes. Exit 0. Green.
That's exactly what happened to us. The eight failures were caught, logged, and skipped, precisely as the code was written to do. The log line was sitting there the whole time, patient, waiting for someone to come looking. Nobody reads logs until there's already a reason to, and our reason arrived two days late with a ticket number on it.
The usual tricks, and why they don't hold up
When you hit this, there are three obvious moves. I've done one of them for years and watched colleagues do the other two.
Fail the whole run if anything fails. Now a 92%-good run pages someone at 3 a.m. They look, see 92 emails went out fine, grumble, and go back to sleep. Do that three times and the alert gets muted. Congratulations, you've trained your team to ignore the one signal you have.
Call it a success and log the details. This was us. The errors were technically recorded. They were also technically invisible, because a log line nobody is searching for is the same as no log line at all. You find it after the ticket. Never before.
Stuff the details into the heartbeat URL. ?failed=8&reason=... Feels clever for about a day. Then a reason string breaks the query, or you hit the length limit, or the whole thing ends up in an access log where, again, nobody is looking. And now your heartbeat is carrying data it was never meant to carry.
None of these answer the actual question. When the job knows something went partly wrong, how does it tell a human, right then, with enough detail that nobody has to go digging?
The fix: let the job tell you
That question is why we built custom alerts into cleverqa.com. I wanted them for myself first; the product part came after.
Think of a custom alert as a log line that goes somewhere people will actually see it. Your code POSTs a small JSON body to a trigger URL. CleverQA turns it into an alert, files a defect, and notifies whoever the service is wired to. No log to open later. No grep. The job tells you what went wrong, in the words you chose when you wrote the error branch.
Setup is one step. Create a custom alert service at cleverqa.com, link it to a project, copy the token from its Trigger tab. Then, in the error branch of the job:
curl -X POST https://monitor.cleverqa.com/api/custom-alert \
-H "Authorization: Bearer $CLEVERQA_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"key": "notification-emails",
"severity": "error",
"title": "8 of 100 notification emails failed to send",
"details": {
"attempted": 100,
"sent": 92,
"failed": 8,
"provider_errors": ["550 mailbox unavailable", "421 try again later"]
}
}'Every field is optional. title is the one-liner you'll see on the alert and the defect. details is whatever JSON you want kept with it: counts, the IDs that failed, the exact strings the provider returned. Put in what you'd want in front of you at 9 a.m. with a coffee, instead of what you'd have to reconstruct from three log files.
About key. A key is just a name you pick for this kind of problem: notification-emails, orders-sync, nightly-export. Letters, digits, dots, dashes, underscores, up to 64 characters. You don't register it anywhere. The first time CleverQA sees a new key it opens an alert and files a defect in the linked project. Every time the same key fires after that, it lands on the same alert as another occurrence instead of opening a new one. That single decision is what stops this from becoming a second noisy log you also don't read.
(If you'd rather skip the header, the token can go in the path: POST https://monitor.cleverqa.com/api/custom-alert/<token>. Same body.)
What happens when it fires
The job sends the alert and gets on with its life. On our side:
The first notification arrives within about 30 seconds. Not on the next digest, not at the top of the hour. If the job runs at 6 a.m. and someone is at their desk, they know before the first ticket exists. For me that's the whole point. I'd rather hear it from the server than from a customer.
You don't get a hundred notifications. If the same key keeps firing, CleverQA sends at most one summary every 10 minutes, and only when there have been new occurrences. The alert records every single one; your phone does not. Acknowledge the alert and the summaries stop.
The payload never leaves CleverQA. Notifications carry the service name, a link, and (in summaries) how many new occurrences there were. The details live in the alert at cleverqa.com, where you go to look at them. Payloads are where secrets accidentally end up, and even though CleverQA redacts the obvious ones before storing anything (passwords, tokens, authorization headers, card-shaped numbers), a Slack channel is still not where a customer's email address belongs.
One defect per key. The defect is filed in your project like any other. If someone closes it there and the same key fires again later, it reopens as a regression instead of creating a duplicate. If it's still open, the new occurrence is simply recorded on it. Nothing auto-closes because the next run happened to be clean. A person decides when it's fixed, which is how it should be.
Heartbeat and custom alert: you want both
A heartbeat answers one question: did it run? If the ping doesn't arrive on schedule, something stopped the job from finishing. Crashes, hung processes, a server that never came back after a deploy, a cron entry that quietly disappeared. The heartbeat catches all of those, and I'd never run a scheduled job without one.
A custom alert answers a different one: did it go wrong? The job ran, finished, pinged the heartbeat, and still has something to confess.
Each one is blind to what the other sees. A heartbeat can't see a run that completed with eight failures inside it. A custom alert can't see a job that never started, because the code that would have sent the alert never executed. Put the heartbeat ping at the end of the job and the custom alert in the error branch, and you've covered both halves. That's the setup I wish I'd had the night those 92 emails went out.
Our email job does exactly that now. The alert has fired twice since, both times for a handful of bounced addresses, and both times we'd dealt with it before anyone outside the team noticed. No tickets. I'll take that.
Try it on your own job
If you have a job that exits 0 every night and you can't say for certain that every record it touched went through, start there. You probably already have one in mind. Add a heartbeat, put a custom alert in the error branch with the real numbers in details, run it once, and see what shows up. It might be nothing. It might be eight.
CleverQA has a free tier, no card required. Start free →