This winter, or rather, one of the days between Catholic Christmas and the New Year, Veeam technical support engineers were busy with unusual tasks: they were hunting for a group of hackers called “Veeamonymous”.
About how the guys themselves came up with and conducted a real quest in reality at their work, with tasks “close to combat,” he told Kirill Stetsko, Escalation Engineer.
"Why did you do this at all?"
- About the same as what people came up with back then Linux – just for fun, for one's own pleasure.
We wanted to move, and at the same time we wanted to do something useful, something interesting. Plus, it was necessary to give some emotional relief to the engineers from their everyday work.
- Who suggested it? Whose idea was it?
- The idea was our manager Katya Egorova, and then the concept and all further ideas were born by joint efforts. Initially, we thought of doing a hackathon. But during the development of the concept, the idea turned into a quest, after all, a technical support engineer is a different kind of activity than programming.
So, we called friends, comrades, acquaintances, different people helped us with the concept - one person with T2 (the second line of support is Ed.), one person with T3, a couple of people from the SWAT team (fast response team for especially urgent cases - Ed.). We all got together, sat down and tried to come up with tasks for our quest.
— It was very unexpected to find out all about this, because, as far as I know, usually quest mechanics are worked out by scriptwriters, that is, not only did you deal with such a complex thing, but also in relation to your work, to your professional field of activity.
- Yes, we wanted to make not just entertainment, but to "pump" the technical skills of engineers. One of the tasks in our department is the exchange of knowledge and training, but such a quest is a great opportunity to let people “touch” some new techniques for them live.
How did you come up with assignments?
- Brainstormed. We had an understanding that we should make some technical tests, and such that they were interesting and at the same time carried new knowledge.
For example, we thought that people should be given the opportunity to try sniffing traffic, use hex editors, and do something for Linux, some slightly deeper things related to our products (Veeam Backup & Replication and others).
Also important was the concept. We decided to start from the theme of hackers, anonymous access and the atmosphere of secrecy. The Guy Fawkes mask was made a symbol, and the name came by itself - Veeamonymous.
"In the beginning was the word"
To stir up interest, we decided to organize a PR campaign on the topic of the quest before the start of the event: we hung posters with an announcement around our office. And a few days later, secretly from everyone, they themselves painted them with spray cans and launched a “duck”, they say, some attackers ruined the posters, even attached a photo with a proof ....
- So you did it yourself, that is, the organizing team ?!
- Yes, on Friday, at 9 o'clock, when everyone had already left, we went and drew the letter "V" in green from the balloons.) Many participants in the quest did not guess who did it - people came up to us and asked who ruined the posters ? Someone very seriously approached this issue and arranged a whole investigation on this topic.
For the quest, we also wrote audio files, “ripped out” sounds: for example, when an engineer logs into our [production CRM] system, there is an answering machine robot that says all sorts of phrases, numbers ... Here we are from those words that he has recorded, composed more or less meaningful phrases, well, maybe a little crooked - for example, we got "No friends to help you" in an audio file.
For example, we represented the IP address in binary code, everything, again, with the help of these numbers [pronounced by the robot], all sorts of frightening sounds were added. We filmed the video ourselves: on the video we have a man sitting in a black hood and wearing a Guy Fawkes mask, but in fact there is not one person, but three, because two are standing behind him and holding a “background” from a blanket :).
“Well, you’re confused, to be honest.
Yes, we are on fire. In general, at first they came up with our technical tasks, and then they composed a literary and gaming canvas on the topic of what supposedly happened. According to the scenario, the participants were hunting for a group of hackers called "Veeamonymous". The idea was also that we, as it were, “break the 4th wall”, that is, we transfer events into reality - here we drew from a spray can, for example.
With the literary processing of the text, one of the native English speakers from our department helped us.
“Wait, why do you need a native speaker?” Did you do it all in English too?!
— Yes, we held for the St. Petersburg and Bucharest offices, so everything was in English.
For the first experience, we tried to make everything just work, so the scenario was linear and quite simple. Added more entourage: secret texts, ciphers, pictures.
We also used memes: there were a lot of pictures on the topics of investigations, UFOs, some popular horror stories - some teams got distracted by this, tried to find some hidden messages there, apply their knowledge of steganography and other things ... but, of course, there is nothing like that was.
About thorns
However, in the process of preparation, we also encountered tasks that were unexpected for ourselves.
They fought a lot over them and solved all sorts of suddenly arising issues, and about a week before the quest they generally thought that everything was gone.
Probably, it is worth talking a little about the technical basis of the quest.
Everything was done on our internal ESXi lab. We had 6 teams, so we had to allocate 6 resource pools. So, for each team, we deployed a separate pool with the necessary virtual machines (same IPs). But since all this was on servers that lie on the same network, the current configuration of our VLANs did not allow isolating machines in different pools. And, for example, during a test run, we got situations where a machine from one pool connected to a machine from another.
How could you fix the situation?
- At first, we thought for a long time, testing all sorts of options with permissions, separate vLANs for machines. As a result, they did this - each team sees only the Veeam Backup server, through which all further work takes place, but does not see the hidden subplot in which they are:
- some Windows machines
- Windows core server
- car with Linux
- pair of VTL (Virtual Tape Library)
All pools are assigned a separate group of ports on the vDS switch and their own Private VLAN. Such double isolation is just needed to completely exclude the possibility of network interaction.
About the brave
- Anyone could take part in the quest? How were the teams formed?
— It was our first experience of holding such an event, and the capacity of our laboratory was limited to 6 teams.
First, as I said, we conducted a PR campaign: using posters and mailing lists, we announced that a quest would be held. We even had some clues - phrases in binary code were encrypted on the posters themselves. In this way, we got people interested, and people themselves agreed among themselves with friends, with buddies, and cooperated. As a result, more applicants responded than we had pools, so we had to make a selection: we came up with a simple test task and sent it to everyone who responded. It was a logic puzzle, it had to be solved quickly.
A team was allowed to have up to 5 people. A captain wasn't required; the idea was cooperation and communication among each other. Someone was strong, let's say, in LinuxSomeone was skilled at tapes (backups on tapes), and everyone, seeing the task, could contribute their efforts to the overall solution. Everyone communicated with each other and found a solution.
- And at what point did this event start? Did you have any "hour X"?
- Yes, we had a strictly appointed day, we chose it so that there was less workload in the department. Naturally, team leaders were informed in advance that such and such teams were invited to participate in the quest, and they needed to be given some relief [regarding loading] on that day. Everything seemed to indicate that it should be the end of the year, December 28, Friday. We expected to take about 5 hours, but all the teams did it faster.
- Everyone was on an equal footing, did everyone have the same tasks based on real cases?
- Well, yes, each of the compilers took some stories from personal experience. We knew about something that this could be in reality, and it would be interesting for a person to “feel” it, look, figure it out. They also took some more specific things - for example, data recovery from damaged tapes. Some with hints, but most teams managed on their own.
Or it was necessary to use the magic of quick scripts - for example, we had a story that some kind of “logic bomb” “torn” a multi-volume archive into random folders along the tree, and we had to collect data. You can do it manually - find and copy [files] one at a time, or you can write a script using a mask.
In general, we tried to adhere to the point of view that one problem can be solved in different ways. For example, if you are a little more experienced or want to "get confused", then you can solve it faster, and there is a direct way to solve it "on the forehead" - but at the same time you will spend more time on the task. That is, almost every task had several solutions, and it was interesting which paths the teams would choose. So the non-linearity was precisely in the choice of the solution.
By the way, the most difficult one turned out to be Linux- the task - only one team solved it independently, without hints.
Could you take hints? Like in a real quest??
- Yes, it was possible to take, because we understood that people are different, and those who lack some kind of knowledge could get into the same team, so in order not to delay the passage and the competitive interest did not disappear, we decided that they would hints. To do this, each team was observed by a person from the organizers. Well, we made sure that no one cheated.
About the stars
- Were there any prizes for the winners?
— Yes, we tried to make the most pleasant prizes both for all participants and for the winners: the winners received designer sweatshirts with the Veeam logo and a phrase encrypted in a hexadecimal code in black). All participants received a Guy Fawkes mask and a branded bag with the logo and the same code.
- That is, everything was like in a real quest!
— Well, we wanted to do a cool, adult thing, and I think we succeeded.
- This is true! And what was the final reaction of those who participated in this quest? Have you achieved your goals?
— Yes, many people came up to me later and said they'd clearly identified their weaknesses and wanted to improve them. Some stopped being afraid of certain technologies—for example, dumping blocks from tapes and trying to extract something from them… Some realized they needed to improve. Linux, and so on. We tried to provide a fairly broad range of tasks, but not entirely trivial ones.
Winning Team
"Whoever wants, he will achieve!"
- Did it require a lot of effort from those who prepared the quest?
- In fact yes. But this was most likely due to the fact that we had no experience in preparing such quests, such infrastructures. (Let's make a reservation that this is not our real infrastructure - it just had to perform some game functions.)
For us it was a very interesting experience. At first I was skeptical, because the idea seemed to me even too cool, I thought that it was very difficult to implement. But they started doing it, started to plow, everything started to catch fire, and in the end we succeeded. And there were even almost no overlays.
In total we spent 3 months. For the most part, we came up with a concept, discussed what we could implement. In the process, of course, something changed, because we understood that we didn’t have the technical ability for something to do it. On the go, I had to redo something, but so that the whole canvas, history and logic would not break. We tried not just to give a list of technical tasks, but to make it fit into the story, so that it was coherent and logical. The main work went on for the last month, that is, 3-4 weeks before X-day.
- That is, in addition to your main activity, did you allocate time for preparation?
- We did this in parallel with the main work, yes.
Are you being asked to do this again?
— Yes, we have many requests to repeat.
- And you?
- We have new ideas, new concepts, we want to attract more people and stretch it out in time - both the selection process and the game process itself. In general, we are inspired by the Cicada project, you can google it - this is a very cool IT topic, where people from all over the world unite, start branches on reddit, on forums, they use the translation of ciphers, and solve riddles, and all that.
- The idea was great, just respect for the idea and implementation, because it is really worth a lot. I sincerely wish you not to lose this inspiration, so that all your new projects are also successful. Thank you!
— Yes, but will it be possible to look at an example of a task that you definitely will not reuse?
“I suspect we won't reuse any of them. Therefore, I can tell about the course of the entire quest.
Bonus trackAt the very beginning, players have the name of the virtual machine and credentials from vCenter. Logged into it, they see this machine, but it does not start. Here you have to guess that something is wrong with the .vmx file. After downloading it, they see the hint needed for the second step. In fact, it says that the database used by Veeam Backup & Replication is encrypted.
After removing the prompt, uploading the .vmx file back and successfully turning on the machine, they see that one of the disks does indeed contain a base64-encrypted base. Accordingly, the task is to decrypt it and get a fully functional Veeam server.
A little about the virtual machine on which all this takes place. As we recall, the quest's protagonist is a rather shady character and is involved in something decidedly less than legal. Therefore, his work computer needed to have a fairly hacker-like appearance, which we had to create, despite the fact that it was... WindowsFirst, they added a bunch of fake stuff like information about major hacks, DDoS attacks, and the like. Then they installed all sorts of typical software and scattered various dumps, hash files, and so on everywhere. It was like in a movie. Among other things, there were folders named "closed-case***" and "open-case***."
To progress further, players need to restore hints from backup files.
It should be noted here that at the beginning, players were given very little information, and most of the data (such as IP addresses, logins, and passwords) was obtained during the quest, finding clues in backups or files scattered across machines. Initially, the backup files were located on Linux-repositories, but the folder itself is on server mounted with a flag noexec, so the agent responsible for file recovery cannot start.
Once the repository is repaired, members have access to all content and can finally restore any information. It remains to understand which one. And for this, they just need to study the files stored on this machine, determine which of them are “broken” and what exactly needs to be restored.
At this point, the scenario shifts away from general IT knowledge towards specific Veeam features.
In this particular example (when you know the file name but don't know where to look for it), you need to use the search function in Enterprise Manager, and so on. Ultimately, after restoring the entire logical chain, the players have another login/password and nmap output. This leads them to Windows Core server, and via RDP (so that life doesn’t seem like a bed of roses).
The main feature of this server: with the help of a simple script and several dictionaries, an absolutely meaningless structure of folders and files was formed there. And when you login, you get a welcome message like “A logic bomb exploded here, so you will have to piece together hints for further steps.”
The next hint was divided into a multi-volume archive (pieces 40-50) and randomly distributed into these folders. Our idea was that players should show their talents in writing simple PowerShell scripts in order to put together a multi-volume archive using a known mask and get the desired data. (But it turned out like in that joke - some of the subjects turned out to be unusually physically developed.)
The archive contained a photo of the cassette (with the inscription "Last Supper - Best Moments"), which hinted at the use of an attached tape library, where there was a cassette with a similar name. Here's just one trouble - it turned out to be so inoperable that it was not even catalogued. Here began, probably, the most hardcore part of the quest. We erased the header from the cassette, so in order to restore data from it, you just need to dump the “raw” (raw) blocks and view them in a hex editor to find file start markers.
We find the marker, look at the offset, multiply the block by its size, add the offset, and using the internal tool, we try to restore the file from a certain block. If everything is done correctly and the math agrees, then the players have a .wav file in their hands.
In it, with the help of a voice generator, among other things, a binary code is dictated, which is revealed in another IP.
This, it turns out, is a new Windows server, where everything hints at the need to use Wireshark, only it is not there. The main trick is that two systems are installed on this machine - only the disk from the second is disabled offline through the device manager, and the logical chain leads to the need to reboot. After that, it turns out that by default a completely different system should boot, where Wireshark is installed. And all this time we were on the secondary OS.
There is nothing special to do here, it is enough to enable capture on a single interface. A relatively close examination of the dump reveals a clearly left packet sent from an auxiliary machine at regular intervals, which contains a link to a youtube video where players are asked to call a specific number. The first caller will listen to the congratulations on the first place, the rest - an invitation to HR (just kidding)).
By the way, we are open for technical support engineers and for trainees. Welcome to the team!
Source: habr.com
