Preliminary notice: this post is purely Friday, and more entertaining than technical. You can expect funny stories about engineering screw-ups, tales from the dark side of working with cellular operators, and other lighthearted musings. If I embellish anything, it's solely for the sake of narrative, and if I fabricate, it's all about events long past, causing no harm to anyone. But if you catch any technical or other nonsense, feel free to correct me relentlessly; I've always stood for justice.
Attention, I'm starting without a warm-up!
Backdoor to the yard
In our duty room on the first floor, there were large windows, almost floor-to-ceiling. They overlooked the staff parking lot, from where various measurers and other field employees departed in the mornings. The parking was sufficiently distant from the main entrance and all service doors, behind two barriers.
One morning, police cars, then still police, pull up to the building, officers stand at all entrances, and start inspecting everyone leaving. An alert hits the internal distribution: unexpectedly (truly unexpectedly, not as usual) a software license inspection has arrived, and workstations will be checked. If anyone has any pirated software on their computers — it needs to be removed immediately!
Of course, regarding operating systems, office, and service software — most of it was licensed. But not everything, not always, and not everywhere; and what the employees installed on their work laptops is a completely dark history. I hurried to check the machines under my responsibility for piracy, quickly removing some...
... Meanwhile, engineers start entering the duty room hurriedly and nervously, holding laptops and systems in their arms. They come in through the door but exit, laughing at the absurdity of the situation, through the window: all the entrances were blocked, but the law enforcement demons didn't think of such a backdoor. So, while the accounting check was occurring (where everything was exemplary), the employees retrieved all the prohibited items.
The past — there
If you're interested and haven't closed the tab, here's some exposition on what's happening in terms of time, space, and characters. I am a wonderfully young IT graduate, as green as a sorrel leaf, working in the engineering duty unit of Samara's Megafon (which at that time was still known as MSS Volga). For me, this was the first real encounter with Technology with a capital T and Technicians with an even bigger T: being the youngest little demon in this hellish kitchen, I watched in awe as experienced engineer demons worked, unsuccessfully trying to grasp their wisdom. Until that wisdom seeped into my brain, I could only stare at a pile of various monitoring systems, getting anxious every time I saw something turn 'red'.
If any of the characters mentioned here recognize themselves, hello to you!
If it works, don't touch it (but do touch it if it doesn't).
One of the aforementioned super-techies was Misha Basov. Over the years working at Megafon, I've heard a lot of good and interesting things about him regarding the fact that he was almost at the origins and launched a bunch of processes. I didn't get a chance to chat with him properly: we met only briefly in the HR department when I brought my documents, and he was picking his up.
One of the monitoring systems we worked with was written by Misha. I don't quite remember what it monitored, but I know Misha created a temporary solution that quickly became a permanent one. And that's good: much of what true techies create for their own urgent needs turns out just great. That monitoring system also satisfied everyone, operating without any support or maintenance, though no one knew how.
A couple of years after Misha's departure, the monitoring started showing a blank page.
I immediately raised the alarm. The shift supervisor raised the alarm. The sector chief raised the alarm.
The head of the department raised the alarm. The head of the service raised the alarm. The head of the department rang the alarm bells. The IT director of the entire Volga region heard the tolling and immediately called a meeting. He summoned the head of the department. The latter shouted at the head of the service. The service head, not understanding the essence of the problem, called the head of the department. The latter, not getting what had happened, called the section head, who summoned the shift leader. Well, that one shifted the blame onto me.
Once, having swapped shifts, I went to this meeting. A lot was said, the person responsible for monitoring was summoned (we didn't hear anything clear), it was recalled that Basov wrote the monitoring, that it is very important, but that no one understands how it works… It all boiled down to needing to get rid of a non-functional and unclear system, and instead implement a proven solution from a verified vendor.
While all this was being said, I managed to borrow a laptop from someone and get SSH access to that server. I was curious to see what kind of super-cool system the legendary Basov had written.
I log in, and out of habit, I type:
df -hThe command responds with something like:
Filesystem Size Used Avail Use% Mounted on
/var 10G 10G 0G 100% / I cleared the overflow of /var/log that had built up over the years, updated the monitoring — everything is working. Fixed it!
The meeting stops, becomes awkward, and everyone disperses. On the way, the head of the department is happy and promises me a bonus!..
… Instead of a bonus, I later received a mental reprimand for accidentally ruining a rollback on the order of the monitoring system from a verified vendor.
Where the little houses live
One of the duties of the duty engineers was to control the electronic access keys to the machine rooms. I was very impressed by the rooms at that time: rows of racks filled with server and switching equipment, fiber optic lines and patch cables (some perfectly laid, some turned into an incredible spaghetti mess), the constant hum of air conditioners, and false floors under which it was so convenient to cool drinks… The entrances to the rooms were sealed with heavy hermetic doors that were designed to ensure automatic locking in case of fire. Entry and exit were strictly logged with signatures, so it was known who and why was currently inside.
What I liked most in these rooms were, of course, the server racks of the 'Superdomes'—two HP SuperDome 9000s that supported billing operations. Two identical nodes, one was always active while the other served as a synchronized hot standby. The only difference between them was their IP addresses; one was x.x.x.45, and the other was x.x.x.46. All engineers were aware of these IPs, because if anything went wrong with billing, the first thing you check is whether the Superdomes are visible. The invisibility of the Superdomes is a major concern.
One morning, such a major concern happens. Within two seconds, all services disappear from both servers, and billing collapses into nothing. We quickly check. server — They ping, but there’s really nothing on them!
We don't even manage to start the necessary set of measures when we hear a loud shout.I'LL KILL YOU, STUDENT!"; the chief admin rushes into the duty room, grabs the electronic key from the shelf and runs there.
Very quickly after this, monitoring returns to normal.
Here’s what happened: a new employee from a contractor, configuring a batch of new virtual machines, manually assigned them sequential static IPs from x.x.x.1 to x.x.x.100. The 'Student' did not know about the sacred untouchable addresses, and it didn’t even occur to the veterans that someone could encroach on them like this.
Anti-Spam Service
Ah, the night shifts! I loved and hated them because it was 50/50: either scheduled maintenance on the equipment where you actively participated, using sleepy brains and trembling hands to assist the engineer, or silence with peace. Clients are asleep, the equipment is working, nothing is breaking down, the duty officer is relaxed.
The duty shift is going as planned.
Once, such midnight peace was disrupted by a call to the service phone: hello, this is Sberbank calling, your SIM card, from which our notifications are sent, has stopped working.
This happened a long time ago, even before the implementation of IP connections to the SMS gateway. Therefore, in order for Sber to send an SMS from their famous number 900, they took the provided SIM card (likely even more than one), plugged it into a GSM modem, and just worked that way.
Okay, I've acknowledged the problem and started digging. First thing, I check the SIM status in the billing system, and it’s blocked. What the hell — there’s a red message next to it stating 'DO NOT BLOCK' and a link to the directive from the supreme archdemon. Wow, this is getting interesting.
I check the reason for the blockage, furrowing my brows and heading to the neighboring office, where a girl from the fraud department is staring at her monitor.
"Lenochka," I say to her, "Why did you block Sberbank?"
She looks confused, saying there was a complaint that spam was coming from number 900. So, I blocked it; we would sort it out in the morning.
And you say — subscriber complaints are ignored!
Of course, the SIM was reactivated.
A very scary story.
When I first started working, they gave me and the other newcomers something like an introductory tour. They showed us the equipment: servers, air conditioning units, inverters, and fire suppression systems. They presented a base station located in one of the machine rooms for experiments, explaining that although the transmitters operate at minimal power, it's better not to enter during that time through the shielded door. They explained how the mobile network works, about primary and backup power, about fault tolerance, and that the network is designed to function even after a nuclear strike. I don’t know if it was said for dramatic effect or true, but it stuck with me.
And indeed: no matter how chaotic things got locally at times, the Volga voice network always worked seamlessly. I'm not a network engineer, but I know that the equipment (both base stations and client terminals) is designed for maximum 'voice' survivability. Power goes out on the base station? It reduces power, switches to diesel generators/batteries, disconnects packet traffic, but calls will still go through. Cut the cable? The base will switch to radio channels, which are sufficient for voice. The phone lost the base station? It will increase power and scan the airwaves until it connects to a tower (or until the battery drains). And so on, and so forth.
But one day, the office lights flickered, and outside, diesel generators started rumbling. Everyone rushed to check their hardware: nothing critical happened on the IT side, but from the monitoring of base stations came a puzzled exclamation of 'uh-oh.' And then: 'Guys, all our base stations are down, check the connection.'
We pull out our phones — there’s no signal.
We're trying out VoIP — there’s no way to reach mobile connections.
There’s no network. At all. Anywhere.
Remembering the words about an atomic bombing, I subconsciously waited for a few seconds, expecting the shockwave to hit us — I couldn't think of any other reason for the network failure. It was both terrifying and curious at the same time: I somehow understood that there was nothing I could do. The other guys were also stunned, no one could figure anything out.
There was no shockwave. After five seconds of shock, we rushed to the wired phone in such cases, starting to call the regional offices. Thankfully, the city network was working, but in the regions, they confirmed: all of Samara is 'dead', no equipment is pinging, and calls aren't going through.
Five minutes later, someone from the power company brought news: there was an explosion somewhere at the power plant, cutting off power to at least all of Samara, possibly even the region. We exhaled; and when the switch to backup power occurred, we even inhaled.
Another terrifying (but a bit silly) story
The biggest fiasco in my memory happened during another live call with the now-defunct leader. At that time, they were introducing the feature of sending questions via SMS, so they prepared in advance for the surge in network load: everything was double-checked and prepared, and a whole week before day X, they banned any work except for emergencies. Such a protocol is activated in any cases when increased loads are expected, for example — during holidays. And for the on-call engineers, it’s the same as a day off because when equipment isn't touched, nothing can happen to it, and even if something happens — all the specialists are sitting in the office just in case.
In general, we sit, listen to the national leader, and don’t worry about anything.
From the switch operators, a quiet 'F***' is heard.
I look over — indeed 'f***': the campus network has dropped.
In just a second, everything died (back then, the meme about Natasha and the cats didn’t exist, but it would have been useful). The user segment of the network disappears, along with the technology segment. With increasing dread, we try to check what remains operational, and after verifying, we reach for the cabinet to grab the hidden bottle of medicinal cognac: only voice calls remain (I told you they are resilient!), everything else has perished. There’s no internet — neither subscriber GPRS nor optical fiber, which is handled by several sub-providers. SMS cannot be sent. It’s a disaster! We call the regions — they have network access, but they can’t see Samara.
Within half an hour, the end of the world became almost physically palpable. Ten million people suddenly found themselves with everything broken and unable to reach the call center because the voice terminals in the call center operate via VOIP.
And this happened during a speech by the darkest ruler! Another victory for the State Department and Obama personally!
The on-duty technicians took off from the starting line and worked very effectively: within an hour, the network came back to life.
Such a screw-up is not just a regional event; it is supposed to be reported to Moscow with all the details and naming the guilty parties. Therefore, those who participated in the investigation were forbidden to tell the truth under the threat of dismissal, and a report full of fluff and vagueness was concocted for official use, making it seem like 'it just happened, no one is to blame.'
What actually happened: one of the bosses had tight deadlines for implementations and bonuses for them were at risk. And his boss's bonuses were also at stake, and so on; thus, they pressured one of the new engineers, instructing him to carry out the required network connections 'while everything is quiet.' The engineer didn’t dare to object, or even ask for a written order: that was his first mistake. The second — he made an error during the remote configuration of the Cisco, achieving record results in failure in no time.
As far as I know — no one was punished.
The holiday is coming to us
Holidays, as I mentioned, have always been special days for us. On such days, the network load increases dramatically, and the number of congratulatory calls and SMS messages skyrockets. I don't know how it is now, with the development of online communication, but back then, on just New Year's Eve, telecom operators would see a significant surge in congratulatory calls.
Therefore, on New Year's Eve, engineers from all departments would be on duty in the office (and outside the office, teams were ready to wade through the snowdrifts to address emergencies at base stations in remote villages). Billing specialists, hardware admins, software technicians, network engineers, switchers, service staff, and support contractors—every type of specialist was present. If conditions allowed, they would gather in our duty room, monitoring the spikes in traffic across the time zones throughout the Volga region.
Three or four times during the night, we welcomed the New Year, though it was less about celebration and more about anxious anticipation: would the equipment withstand the overload, would any link in the complex technical chain break…
Sasha, who was responsible for billing, was especially nervous. He always seemed like he was living on edge, as he had to deal with everything happening with billing, answer for all the mistakes, and was often the one woken up at night; honestly, I can't imagine how and why he worked where he did. Maybe he was paid a lot, or his family was being held hostage. But that night, I had the feeling that if you snapped your fingers at Sasha, the accumulated stress inside him would turn him to dust. In such a situation, we have a broom, but for now, we worked, eyeing the cognac waiting for its turn.
Hour by hour, all the traffic spikes passed, and everyone began to double-check their systems. The switcher paled: on one of the regional switches, all billing traffic disappeared. This means data about all the calls that passed through the switch; it's written to a file, which is transferred chunk by chunk via FTP (it's clunky but reliable) to the BRT for billing.
The operator, imagining the volume of the turpentine enema he would receive for losing part of the New Year's revenue across an entire region, actually trembled. Turning to Sasha, he addressed the esteemed billing gentleman with a voice full of anxious hope: "Sasha, please take a look, maybe BRT has managed to extract the billing? Oh, please check!".
Sasha sipped some cognac, had it with a caviar sandwich, chewed slowly, and, rolling his eyes in pleasure from the fact that the screw-up wasn't his, replied: "I've already checked, there are no files...".
(My wonderful editor asked what happened to the poor operator afterwards. Oh, his fate was terrible: he was sentenced to a week of shifts on the front line of the call center support, forbidden to curse. Brrr!)
Throw a stone, whoever is without sin
From these stories, one might get the impression that neither I nor the other operators ever messed up. Nothing of the sort; we messed up, but somehow without any interesting epic or consequences. The job was considered suitable for yesterday's students without brains or experience, and firing such an employee didn't guarantee a smarter replacement. However, shifting one's mistakes onto the 'duty shift' was a separate sport for engineers: let it slide, failed to clarify, didn't notify on time, and then they got punished. The 'duty' had mastered the art of making excuses perfectly; it didn't always work, but everyone understood each other. So, punishment came — but usually without serious consequences.
We are analyzing another 'gaffe' during the shift change.
In the several years I worked there, I can recall three instances when someone was fired from the department.
Once, an engineer on the night shift decided to crack open a beer, and suddenly the technical director walked into the duty room. He sometimes could just drop by to say hello (as if he started out in duty himself). He caught the guy with a can of beer, clicked on his phone, and there was a dismissal. No more beer during the night shifts after that.
Another time, the duty switchboard operator missed a seriously bad accident. I don't remember the details anymore.
And the third time — already towards the end of my work there. The working conditions had deteriorated significantly, there was a huge turnover, and terrible overtime. People sometimes worked for a whole day, then would sleep for 12 hours and go out for another day shift. I worked like that myself as long as my health allowed and it was paid; then they practically stopped compensating for the overtime (they standardly promised compensation in time off when there was an opportunity — but everyone understood that no one would ever take that time off), and they would send us to shifts almost under threats. One engineer lost it; in the middle of his shift, he got up from his workstation and left home forever, stopping by the chief's office to tell him off. I remember an email where this engineer was retrospectively branded a fascist and a traitor; every line was filled with how the management was furious about his actions.
Regarding my personal screw-ups — one incident stood out due to its unusual nature. Again, it was a night shift, everything was quiet, nothing was happening. During the shift change, we checked the monitoring: oops, the data processing from the switches had dropped overnight, the red light had been on for quite a while. I had been watching this signal all night — yet I didn’t seem to acknowledge it at all. Despite the fact that it was one of the most obvious and clear monitoring systems, I still don't understand why I didn't see it.
Here, there were no excuses to come up with, it was a clear and straightforward mistake, a fifth-category accident, and quite possibly a dismissal. After twelve hours of night duty, they kept me until lunch, forcing me to write explanations. Since in truth no one would believe me, I had to come up with some nonsense, claiming that I had taken too much pain relief due to an injury and fell asleep. The head of the department shouted at me in his office; overall, everything was heading toward dismissal — but it resulted in a reprimand with a pay cut. Bonuses had not been seen at Mega for several years by then, so I didn't suffer any actual loss.
Remembering the episode with the arrival of the tech director: one night, a certain jerk barged into the duty room and started yelling that we were unguarded (the duty room shouldn't be locked in principle), that we were all fools, and that by morning he expected explanations from us for all our screw-ups. This jerk was the head of security, and he was quite intimidating. After yelling, he left into the darkness, and in the morning we asked our boss—what should we do? "Just tell him to f**k off," he replied, and that settled the incident.
How I Broke the Department
Back in those days, bash.org (then still bash.org.ru, not what it is now) was a cult resource. Quotes appeared there almost a couple of times a month, and having YOUR OWN! QUOTE!!! ON BASH!!! was as cool as having your second-level domain back in the 2000s. That bash.org was more IT-anime oriented, but it was funny for everyone.
Every working morning of the youngest engineer (that is, mine) started with reading bash.org—a thirty-second laugh before twelve hours of suffering.
One day a colleague asked me what I was giggling about. I showed him. He forwarded the link around the department.
Work came to a halt for a couple of days: to my surprise, none of my colleagues knew about bash until that moment. The duty room was filled with laughter: "Ah-haha-haha, patching KDE, ahaha-haha!". "Hee-hee-hee, sinking weights in mercury, bgegege!". The workday was lost, but on the other hand, we prolonged our lives quite a bit.
Bonus for those who read to the end
Remember back in the day there was a popular joke, "I see two C drives in Norton, I think—why do I need two? So I deleted one!". It closely resembles one of my favorite stories that I don't tell, but it gets told to me. And it's funny every time, just like the first time:
18+, but you can't take words out of a song
Postscript
These stories are a processed compilation of some posts from my TG channel. Sometimes similar nonsense pops up; I'm not hinting at anything, but Have a good non-f*ked-up Friday, everyone!
Preliminary notice: this post is purely Friday-related and more entertaining than technical.
Source: habr.com
