🥇We are looking for the problem in the wrong place

This is a short story from real practice, where a small problem, well-disguised by redundancy, turns into a headache.

A brief disposition:

A small branch office, equipped with its own PBX (Asterisk + FreePBX) running on desktop hardware, and a local terminal server with 1C, a file storage, and a virtual domain RO controller. The internet is distributed via Mikrotik. It's a small branch, and this is enough for them.
It all started with monitoring (due to lack of time and laziness, not everything gets monitored) that reported overheating of one server (with PBX) at the branch. While the locals were solving the issue, the old man hung up and damaged the MySQL database a bit.

Many signs pointed to trouble, but not this one...

No worries, the database was fixed, everything should work. But the locals complain that calls are getting dropped. Alright — issues can happen in FreePBX, I take a backup, restore it, and everything is fine.
But the trouble remains; locals still complain that calls aren't going through properly. They say calls seem to go through normally, but when they call out or call each other, there is a delay of several seconds. I start looking through the large and puzzling logs of Asterisk and FreePBX, but cannot identify the problem. I remember there was an issue with STUN and ICE that caused a similar delay. I disable everything entirely, and the result is zero.

Despair — a path to making bad decisions:

I fall into despair, hours of fiddling with the PBX yield no good results; it's already deep into the night and the problem is unresolved.
I leave the problem for the morning, hoping for a fresh perspective. The next morning, another unfortunate decision is made: since the system failed (even though the hang-up couldn't have been that destructive), I try to fix the system by reinstalling all packages. The result is just slightly better; the delay has decreased (not significantly, but it’s already a success).
I make yet another bad decision: if partial repairs of the OS (and database restoration from backup) had a little success, but the root of the problem is still unclear, and I have already spent a lot of time searching for the cause, I decide to act radically: we wipe the OS and install everything anew (thankfully, the process automation makes it feasible in a reasonable time). I roll back the FreePBX configuration from a backup. Another failure. The result is zero!

Despair — reason clouds, and decisions get even worse.

I'm sinking into despair. Really bad thoughts start to come up, and I think: maybe the config in the backup is corrupted (I had this happen after a series of updates where things stopped working, and I could never find the cause). There's nothing left to do: I have to reinstall everything from scratch by hand. What a shame! The result is strictly zero, and I've wasted a lot of time!

Acceptance is the path to awareness

In desperate attempts to understand what’s happening, I start closely examining the logs. I notice a pattern. The call to Extension happens exactly at 5 seconds, while for a group of calls with 3 Extensions, it’s 15 seconds! I begin to Google about call delays, specifically indicating my exact delay. I stumble upon an answer that I have already found; people say the problem lies in DNS, but I know for sure there’s no issue, all addresses resolve correctly!

The obvious is not the probable

There's nothing to do; I grab nslookup and bingo (I should have done this right away)! The primary DNS is down (a virtual machine with the controller), and I didn't even notice! If there had been just one DNS, there would have been an error right away 😉

Summary

An elementary problem that monitoring (which should be set up for all nodes) could have detected, masked by DNS redundancy, led to a loss of almost two working days trying to solve a silly situation. Laziness creates a headache; setting up monitoring takes a minute—looking for a problem where there isn’t one takes two days.

Only registered users can participate in the survey. Please log in, please.