{"id":32682,"date":"2019-10-31T21:48:22","date_gmt":"2019-10-31T18:48:22","guid":{"rendered":"https:\/\/prohoster.info\/blog\/ishhem-problemu-ne-v-tom-meste\/"},"modified":"2019-10-31T21:48:22","modified_gmt":"2019-10-31T18:48:22","slug":"ishhem-problemu-ne-v-tom-meste","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/ishhem-problemu-ne-v-tom-meste","title":{"rendered":"We're looking for the problem in the wrong place","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

This is a short story from real practice, where a small problem, well-disguised by redundancy, turns into a headache.
\n<\/a><\/noindex><\/p>\n

A brief disposition:<\/h2>\n

\nA small branch office, equipped with its own PBX (Asterisk + FreePBX) running on desktop hardware, and a local terminal server with 1C, a file storage, and a virtual domain RO controller. The internet is distributed via Mikrotik. It's a small branch, and this is enough for them.
\nIt all started with monitoring (due to lack of time and laziness, not everything gets monitored) that reported overheating of one server<\/a> (with PBX) at the branch. While the locals were solving the issue, the old man hung up and damaged the MySQL database a bit.<\/p>\n

Much was ominous, but not this...<\/h3>\n

\nNo worries, the database was fixed, everything should work. But the locals complain that calls are getting dropped. Alright \u2014 issues can happen in FreePBX, I take a backup, restore it, and everything is fine.
\nBut the trouble remains; locals still complain that calls aren't going through properly. They say calls seem to go through normally, but when they call out or call each other, there is a delay of several seconds. I start looking through the large and puzzling logs of Asterisk and FreePBX, but cannot identify the problem. I remember there was an issue with STUN and ICE that caused a similar delay. I disable everything entirely, and the result is zero.<\/p>\n

Despair \u2014 a path to making bad decisions:<\/h3>\n

\nI fall into despair, hours of fiddling with the PBX yield no good results; it's already deep into the night and the problem is unresolved.
\nI leave the problem for the morning, hoping for a fresh perspective. The next morning, another unfortunate decision is made: since the system failed (even though the hang-up couldn't have been that destructive), I try to fix the system by reinstalling all packages. The result is just slightly better; the delay has decreased (not significantly, but it\u2019s already a success).
\nI make yet another bad decision: if partial repairs of the OS (and database restoration from backup) had a little success, but the root of the problem is still unclear, and I have already spent a lot of time searching for the cause, I decide to act radically: we wipe the OS and install everything anew (thankfully, the process automation makes it feasible in a reasonable time). I roll back the FreePBX configuration from a backup. Another failure. The result is zero!<\/p>\n

Despair \u2014 reason clouds, and decisions get even worse.<\/h3>\n

\nI'm sinking into despair. Really bad thoughts start to come up, and I think: maybe the config in the backup is corrupted (I had this happen after a series of updates where things stopped working, and I could never find the cause). There's nothing left to do: I have to reinstall everything from scratch by hand. What a shame! The result is strictly zero, and I've wasted a lot of time!<\/p>\n

Acceptance is the path to awareness<\/h3>\n

\nIn desperate attempts to understand what\u2019s happening, I start closely examining the logs. I notice a pattern. The call to Extension happens exactly at 5 seconds, while for a group of calls with 3 Extensions, it\u2019s 15 seconds! I begin to Google about call delays, specifically indicating my exact delay. I stumble upon an answer that I have already found; people say the problem lies in DNS, but I know for sure there\u2019s no issue, all addresses resolve correctly!<\/p>\n

The obvious is not the probable<\/h3>\n

\nThere's nothing to do; I grab nslookup and bingo (I should have done this right away)! The primary DNS is down (a virtual machine with the controller), and I didn't even notice! If there had been just one DNS, there would have been an error right away \ud83d\ude09<\/p>\n

Summary<\/h3>\n

\nAn elementary problem that monitoring (which should be set up for all nodes) could have detected, masked by DNS redundancy, led to a loss of almost two working days trying to solve a silly situation. Laziness creates a headache; setting up monitoring takes a minute\u2014looking for a problem where there isn\u2019t one takes two days.<\/p>\n

Only registered users can participate in the survey. Please log in<\/a><\/noindex>, please.<\/p>\n

Has anything similar ever happened to you?<\/h2>\n