A loud failure page is annoying. A failure that produces no signal at all is the one that costs you a customer, because you find out about it after they do.
Rank your worst production incidents by how you found out about them, not by how bad they were. A fair number will be things a support ticket surfaced, or a founder noticed by accident while looking at something unrelated. The common thread in those cases usually isn’t that the failure was subtle. It’s that nothing in the stack was watching for the absence of a thing, only for the presence of an error.
The job that stops without complaint. A cron task that used to run every night starts exiting silently after a dependency update changes a file path. No exception, no non-zero exit code; the script just does nothing and returns 0. Log-based alerting has nothing to grep for, because there’s no error line to match. The only way to catch this is to expect a signal on a schedule and alert when it doesn’t show up, which is a fundamentally different check than “did this request succeed.”
The certificate that expires on a quiet Saturday. Auto-renewal has worked for eighteen months, so nobody thinks about it, until a DNS change breaks the ACME challenge and the renewal starts failing weeks before anyone notices, because the failure itself is also silent unless something is specifically watching the expiry date rather than just pinging the endpoint. The site keeps answering 200 right up until the hour the old certificate runs out.
The endpoint that’s up but wrong. A health check hits /health and gets a 200 back, because the process is alive. Meanwhile the actual feature users care about is returning errors, because a health check tells you the process didn’t crash, not that the product works. This is the gap between “the server is up” and “the thing people are trying to do actually happens,” and it’s wide enough to drive a very embarrassing incident through.
Standard log monitoring is built around pattern matching: watch for ERROR, watch for a stack trace, watch for a spike in 5xx responses. That works when the failure produces output. All three cases above are failures precisely because they don’t. A grep-based alert can only fire on what’s written; it has no way to notice what should have been written and wasn’t.
The fix isn’t a smarter grep. It’s a different kind of check that expects a signal on a schedule (a ping, a certificate expiry date, a specific assertion on a response body) and treats the absence of that signal as the alert, rather than waiting for an error to appear that these failure modes never produce.
For the missing job: a monitor that expects a run/complete pair inside a known interval, with a grace period for jobs whose duration varies, and treats a missed window as a failure state on its own.
For the certificate: a check that reads the expiry date off the full certificate chain on a schedule, not just whether the TLS handshake currently succeeds, with a warning threshold days ahead of the actual deadline so there’s time to fix a broken renewal before it becomes an outage.
For the endpoint that’s technically up: a check that asserts on the response itself, a status code, a string in the body, a response time ceiling, not just whether a connection was accepted.
None of these three checks are complicated on their own. What they have in common is that they all require deciding, in advance, what “nothing happened when something should have” looks like, and building the alert around that absence instead of around an error you’re hoping never to have to grep for.
Sign up, get an API key, paste the curl into your cron. That's it.