Advance notice: this post is purely Friday, and more entertaining than technical. Funny stories about engineering fakups, tales from the dark side of the work of a cellular operator and other frivolous rustle are waiting for you. If I embellish something somewhere, it’s only for the benefit of the genre, but if I’m lying, then all this is the work of the days so past that no one will be harmed by that. But if you catch a technical eye or something else, correct me mercilessly, I have always been on the side of justice.
Attention, I'm starting without overclocking!
backdoor to yard
In our duty room on the first floor there were large such windows, from the basement and almost to the ceiling. They went out to the service parking lot, from where in the morning all sorts of measurers and other field employees drove around. The parking lot was at a sufficient distance from the front and all service entrances, and even behind two barriers.
One morning, police cars drive up to the building, policemen stand at all checkpoints and search all those leaving. An alert arrives in the service mailing list: all of a sudden (really suddenly, not as usual) a software license check came up, workstations will be inspected. Whoever has something pirated on their computers - you have to demolish it right away!
Of course, everything related to operating systems, office and service software was mostly licensed. But not everything, not always and not everywhere; and what the employees set for themselves on office laptops is a completely dark story. I rushed to check the cars in my area of responsibility for piracy, quickly demolishing something ...
... And at this time, engineers begin to enter the duty room with a hurried and nervous step, with laptops and system engineers in their arms. They enter through the door, and go out, giggling at the absurdity of the situation, through the window: all the checkpoints were blocked, but the demons of law and order did not think of such a backdoor. So, while the accounting department was being checked (where everything was exemplary), the employees pulled out everything pale.
The past is there
If you are interested and have not closed the tab, here is some exposition of what is happening in time, space and faces. I am a beautifully young, green as a sorrel leaf, IT graduate who got a job in the engineering duty of the Samara MegaFon (which was then also MSS Povolzhye at all). For me, this was the first real contact with Technique with a capital letter and Techies with an even greater one: being the youngest imp in this hellish kitchen, I watched with delight the work of highly experienced devils of engineers, unsuccessfully trying to comprehend their wisdom. Until that wisdom seeped into the brain pores, I could only poke into a bunch of various monitorings, worrying every time a "red" appeared there.
If one of the characters mentioned here suddenly recognizes himself - hello to you!
Works - do not touch (but touch if it does not work)
One of the super-techies mentioned above was Misha Basov. Over the years of work at Mega, I heard a lot of good and interesting things about him in the spirit that he was almost at the origins and launched a bunch of processes. I didn’t manage to communicate with him properly: we met literally in the personnel department, when I brought the documents, and he took them away.
One of the monitoring systems we worked with was written by Misha. I don’t really remember what was monitored there, but I know that Misha wrote a temporary solution, which quickly became permanent. Yes, and it's good: much of what true techies do for their own needs in a hurry, it turns out just fine. That monitoring also suited everyone, working without any support and maintenance, however, no one knew how.
A couple of years after Misha's dismissal, monitoring began to show a blank page.
I immediately sounded the alarm. The shift supervisor sounded the alarm. The head of the sector sounded the alarm.
The head of the department sounded the alarm. The head of the service sounded the alarm. The head of the department tinkled his bells. The ringing was heard by the IT director of the entire Volga region, who immediately called a meeting. There he called the head of the department. He yelled at the chief of service. He, not understanding the essence of the problem, called the head of the department. This one, not understanding what had happened, called the head of the sector, who called the head of the shift. Well, he turned the arrow on me.
Having somehow changed from duty, I went to this meeting. Many words were said, the person responsible for monitoring was called (we did not hear anything intelligible), it was remembered that Basov wrote monitoring, that monitoring is very important, but that no one understands and does not know how it works ... It all boiled down to the fact that a non-working and incomprehensible system must be removed, and instead a proven solution from a proven vendor should be implemented.
While all this was being said, I begged someone for a laptop and ssh access to that server. It was interesting for me to see what kind of super-cool system the legendary Basov wrote.
I go in, first of all, out of habit, I type:
df -hThe command tells me something like:
Filesystem Size Used Avail Use% Mounted on
/var 10G 10G 0G 100% /
I clean the /var/log that has overflowed over the years, update the monitoring - everything works. Fixed it!
The meeting stops, crumple, everyone disperses. On the way, the head of the department rejoices and promises me a bonus! ..
… Instead of a bonus, I later received a mental stick for the fact that I accidentally broke off a rollback to order a monitoring system from a trusted vendor.
Where do the houses live
One of the duties of the engineers on duty was to control the electronic access keys to the machine rooms. The halls themselves then impressed me very much: rows of racks crammed with server and switching equipment, lines of fiber optics and cross-cables (somewhere - perfectly laid, somewhere - turned into an incredible ball of spaghetti), the constant hum of air conditioners and raised floors, under with which it was so convenient to cool drinks ... The entrances to the halls were clogged with heavy hermetic doors, designed to provide automatic blocking in case of fire. Entry and exit were strictly recorded under the signature, so that it was known who was inside and why.
Most of all, in these halls, I liked, of course, the server cabinets of the "superhouses" - two HP SuperDome 9000, which ensured the billing. Two identical nodes, one was always combat, and the second was a synchronous hot standby. The difference between them was only in IP addresses, one was xxx45, the other was xxx46. Both of these IPs were known to all the engineers, because if something happened on the billing, the first thing you look for is whether the superhouses are visible. The invisibility of superhouses is akhtung.
One morning, a similar akhtung happens. Within two seconds, all services disappear on both servers, billing collapses into nothing. We quickly check the servers - they ping, but there really is nothing on them!
We do not even have time to begin the set of measures, as we hear a loud yell "KILL, STUDENT!"; the archadmin of all servers runs into the duty room, tears off the electronic key to the turbine hall from the shelf and runs there.
Very quickly after that, monitoring returns to normal.
This is what happened: a new employee of the contractor, who was configuring a bunch of new virtual machines, manually assigned them sequential static IP addresses, from xxx1 to xxx100. The "student" did not know about the sacred untouchable addresses, and it never occurred to the old-timers that someone could encroach on them like that.
Antispam service
Wow, night shift! I loved them and hated them, because it was 50/50: either planned work on the equipment, where you take an active part, helping the engineer with sleepy brains and trembling hands, or silence with calmness. The subscribers are asleep, the equipment is working, nothing breaks, the attendant is relaxed.
The work is going according to plan.
Once such midnight calm is broken by a call to the office phone: hello, this is from Sberbank, they are disturbing us, your SIM card, with which our alerts are sent, has stopped working.
It was a long time ago, even before the introduction of IP connections to the SMS gateway. Therefore, in order for Sberbank to send a text message from their famous number 900, they took the provided SIM card (most likely, not even one), stuck it into the GSM modem, and that’s how it worked.
Okay, I accepted the problem and started digging. First of all, I check the status of the SIM card in the billing, it is blocked. What the hell - next to it is a red inscription "DO NOT BLOCK" and a link to the order of the general archdemon. Wow, that's really interesting.
I check the reason for the blockage, make my eyebrows look like a house and travel to the next office, where a girl from the fraud department is staring at the monitor.
“Helen,” I tell her, “why did you block Sberbank?”
She is incomprehensible: they say a complaint came that spam was coming from number 900. Well, I blocked, in the morning they would figure it out.
And you say - subscriber complaints are ignored!
Simka was turned back on, of course.
Very scary story
When I first got a job, I and other newcomers were given something like an introductory tour. They showed the equipment: servers, condos, inverters, fire extinguishing. They showed the base station, which was in one of the turbine halls for experiments, explaining that although the transmitters are switched on at minimum power, it is better not to enter the screened door at this time. They explained about the device of the mobile network, about the main and backup power supply, about fault tolerance and about the fact that the network is designed to work even after the atomic bombing. I don’t know if it was said for a red word, or true, but it stuck in my head.
And indeed: no matter what kind of akhtung sometimes happens locally, the Volga voice network has always worked continuously. I am not a signalman, but I am aware that the equipment (both base stations and client terminals) is designed for maximum survivability of the "voice". Did the power to the BS turn off? It will reduce power, switch to DGU / batteries, turn off the transmission of packet traffic, but the voice will go on. Did you cut the cable? The base will switch to a radio channel, which is enough for a voice. Phone lost BS? He will increase the power and will feel the ether until he catches on the tower (or until he drops the battery). Etc.
But one day the lights flickered in the office, and diesel generators rumbled on the street. Everyone rushed to double-check their pieces of iron: nothing critical happened in the IT part, but from the monitoring of the BS there was a puzzled “panky”. And then: “guys, we have ALL the bases, check the connection.”
We take out mobile phones - there is no signal.
We are trying IP-telephony - there is no way out for mobile communications.
There is no network. At all. Nowhere.
Remembering the words about the atomic bombing, I subconsciously waited for several seconds for the shock wave to reach us - for some reason, another reason for the loss of the network did not occur to me. It was scary and curious at the same time: I somehow understood that I wouldn’t have time to do anything. The rest of the guys were also dumbfounded, no one could understand anything.
There was no blast wave. After a five-second shock, they rushed to the wired city network available for such a case, starting to call regional offices. The city network, fortunately, worked, but in the regions they confirmed: the whole of Samara is “dead”, neither the pieces of iron are pinged, nor is dialing.
Five minutes later, one of the power engineers brought the news: there was a bang somewhere at a power plant, at least the entire Samara was de-energized, and possibly the region. exhaled; and when there was a switch to reserve capacities, they even breathed in.
Another scary (but a bit stupid) story
The biggest fakup in my memory happened during the next straight line with the now zeroed one. At that time, they were just introducing a trick with sending questions by SMS, so they prepared for a surge in the load on the network in advance: they double-checked and prepared everything, and for a whole week before day X, they banned any work, except for emergency ones. A similar protocol is activated in any case when an increased load is expected, for example, on holidays. And for the engineers on duty, it is the same as a day off, because when the equipment is not touched, nothing can happen to it, and even if it happens, all the specialists, just in case, sit in the office in advance.
In general, we sit, listen to the national leader, we don’t worry about anything.
From the side of the switchboards, a low “E *** t” is heard.
I look at myself - really “f *** b”: the campus network has fallen off.
In a second, everything dies in general (then there was no meme about Natasha and cats, but it would be useful). The user segment of the network disappears, the technological segment disappears. With increasing horror, we try to check what is left in working condition, and after checking, we reach for the locker for a hidden bottle of medicinal cognac: only voice calls remain (I told you, they are tenacious!), everything else is dead. There is no Internet - neither subscriber GPRS, nor on optics, which is assigned to several sub-providers. SMS is not sent. Ass! We call the regions - they have a network, but they do not see Samara.
Within half an hour the end of the world became almost tangible. Ten million people who suddenly have everything broken and who cannot get through to the call center because the voice terminals in the call center work through VOIP.
And this is during the speech of the darkest ruler! Another victory for the State Department and Obama personally!
The techies on duty were blown up from a low start and worked very well: within an hour the network came to life.
Such a flight is not at the regional, or even at the regional level, it is necessary to report this to Moscow with all the details and extradite the perpetrators. Therefore, those who took part in the investigation were forbidden to tell the truth under pain of dismissal, and for the Civil Defense they wrote a report full of water and fog, it somehow turned out that “it itself, no one is to blame.”
What actually happened: one of the bosses was running out of time for implementations and breaking off bonuses for them. And the chief of the chief was broken off, and so on; so they put pressure on one of the new engineers, telling him to make the required power ups "while all is quiet." The engineer did not dare to object, or even to demand a written order: this was his first mistake. The second is that he made a mistake with the remote configuration of the cisco, achieving record results in fakap in the shortest possible time.
As far as I know, no one was punished.
The holiday comes to us
Holidays, as I mentioned, have always been special days for us. On such days, the load on the network increases sharply, the number of congratulatory calls and SMS goes off scale. I don’t know how it is now, with the development of Internet communication, but then on New Year’s Eve alone, opsos took off a very significant foam on congratulatory calls.
Therefore, on New Year's Eve, engineers from all departments were on duty in the office (and outside the office - teams ready to push through the snowdrifts to eliminate the accident at the base station in the village of small drischi). Billing specialists, iron admins, software plumbers, networkers, commutators, service providers, support contractors - every creature on the creature. And if conditions allowed, then they hung out in our duty room, watching on our monitoring devices bursts of traffic following the time zones throughout the Volga region.
Three or four times a night we celebrated the New Year, however, this was not so much a celebration as a nervous expectation: whether the equipment will withstand the overload, whether some link in the complex technical chain will break ...
Sasha, who was in charge of billing, was especially nervous. In principle, he always looked as if his whole life was spent on a bare nerve, because he had to rake up all the good things that were happening with billing, be responsible for all the jambs, he was woken up more often than others at night; in general, I have no idea how and why he worked where he worked. Maybe he was paid a lot of money, or the family was held hostage. But that night, I generally had the feeling that if you click on Sasha with a fingernail, then from the internal stress accumulated in him, he will crumble into dust. For such an unpleasant event, we have a broom, but for now we are working, licking our lips at the cognac waiting in line.
Hour after hour, all bursts of load passed, everyone began to double-check their systems. The switch turns pale: one of the regional switches lost all billing traffic. And this is data on all calls that have passed through the switch; they are written to a file, which is downloaded in chunks via FTP (condo, but reliably) to BRT for billing.
The commutator, imagining how much turpentine enema would be given to him for the loss of part of the New Year's revenue for the entire region, already trembled. Turning to Sasha, he turned to the illustrious billing specialist in a voice full of exciting hope: “Sasha, look please, maybe BRT managed to pump out the billing? Oh, look, please!”
Sasha took a sip of cognac, ate it with a caviar sandwich, chewed slowly and, rolling his eyes with pleasure due to the fact that he did not have a joint, answered: “I already checked, there are no files ...”.
(My wonderful proofreader asked what happened to the poor commutator afterwards. Oh, his fate was terrible: he was sentenced to a week of duty on the first line of call center support, forbidding swearing. Brrr!)
Throw a stone who is sinless
According to these stories, one might get the impression that neither I personally nor the other duty officers did not screw up. Nothing like that, fakapili, but somehow without an interesting epic and consequences. The work was considered suitable for yesterday's students without brains and experience, there is nothing to take from such an employee, to kick him out for a jamb - so a new one is not a fact that it will be smarter. But dumping their jambs on the “duty room” was a separate sports discipline for engineers: they missed the mark, didn’t figure it out, didn’t notify them in time, so punish them. The “duty officer” perfectly mastered the science of making excuses, it didn’t always work out, but everyone understood everything. Therefore, they flew in - but, as a rule, without serious consequences.
We analyze the next "flight" at the shift change.
In a few years of working there, I can remember three cases when someone was fired from the department.
One day, an engineer decided to have a beer on the night shift, and then take the technical director to the duty room and come in. He could sometimes come in like this in a simple way to say hello (it seems like he himself started with the attendants). Burned a guy with a can of beer, click on the phone, fired. No more beer at night.
Another time, the switchboard operator on duty missed some very terrible accident. I don't remember the details.
And the third time - already at the end of my work there. Working conditions sank very badly, there was a wild turnover and terrible overtime. People sometimes worked day and night, then went to sleep for 12 hours and again went on daily duty. I myself worked that way as long as my health allowed and it was paid; then they actually stopped paying for overtime (as a standard, they promised compensation with time off when there was an opportunity - but everyone understood that no one would ever go for a walk), and they were kicked out on duty almost with threats. One engineer could not stand the cuckoo, he got up from his workplace in the middle of the shift and went home forever, on the way looking into the office of the head of the service, and sending him three letters. I remember the mailing list, in which this engineer was post factum branded a fascist and a traitor, in every line it was read how the authorities burned from such an act.
With regards to my personal fakapov - one case was remembered for its unusualness. Again, night duty, everything is quiet, nothing happens. At the shift change, we check the monitoring: oops, data processing from the switches fell at night, the red light has been on for a long time. I spent the whole night looking at this signal - and did not perceive it or something. Despite the fact that it was one of the most obvious and visual monitoring, I still don’t understand why I didn’t see it.
There were no excuses to stick here, the joint is clean and one hundred percent, an accident of the fifth category and a very likely dismissal. After twelve hours of night duty, they kept me busy until lunchtime, forcing me to write explanatory notes. Since no one would believe the truth, I had to come up with some kind of babble that I had overate the painkiller due to the injury and fell asleep. The head of the service yelled at me in his office, in general, everything went to dismissal - but resulted in a reprimand with deprecation. By that time, Mega had not seen any bonuses for several years, so I did not suffer any damage.
Recalling the episode with the arrival of the tech director: one night some goon crept into the duty room and started yelling that we were sitting unlocked (the duty room should not be locked in principle), that we were deer here, and that by morning he was waiting for all of us explanatory about all our jambs. This redneck was the head of the security service, and he stank. Having yelled, the chief without dumped into the darkness, and in the morning we asked our boss - they say, what should we do? “Send him the fuck,” he replied, and the incident was over.
How I broke the department
In those days, bashorg (then still bash.org.ru, and not where it is now) was a cult resource. Quotes appeared there almost a couple a month, and have YOUR OWN! QUOTE!!! ON THE BASH!!! was as cool as, say, your second-level domain in XNUMX. That bashorg was somehow more IT-anime, although it was funny for everyone.
Every working morning of the youngest engineer (that is, mine) began with reading bashorg - thirty seconds of laughter before twelve hours of suffering.
One day a colleague asked me what I was giggling about. I showed him what. He sent the link around the department.
The work got up for a couple of days: to my surprise, none of my colleagues knew about bash until that moment. There was laughter in the duty room: “Ah-haha-haha, patch KDE, ahaha-haha!”. "Igogo-go-go, drown crowbars in mercury, bgegegeg!". The working day was lost, on the other hand, then life was extended to itself notably.
Bonus for Readers
Remember, in the bearded days there was such a popular joke “I see two C drives in Norton, I think - why do I need two? Well, erased one! It is very reminiscent of one of my favorite stories, which is not told by me, but by me. And every time it's funny, like the first time:
18+, but you can't throw words out of a song
Postscript
These stories are a processed compilation of some posts from my TG channel. Sometimes there slips similar game; I'm not implying anything, but I'll leave it anyway.
Happy Happy Friday everyone!
Source: habr.com