This winter, or rather, one of the days between Catholic Christmas and the New Year, Veeam technical support engineers were busy with unusual tasks: they were hunting for a group of hackers called “Veeamonymous”.

About how the guys themselves came up with and conducted a real quest in reality at their work, with tasks “close to combat,” he told Kirill Stetsko, Escalation Engineer.

"Why did you do this at all?"

- Approximately the same as people came up with Linux in their time - just for fun, for their own pleasure.

We wanted to move, and at the same time we wanted to do something useful, something interesting. Plus, it was necessary to give some emotional relief to the engineers from their everyday work.

- Who suggested it? Whose idea was it?

- The idea was our manager Katya Egorova, and then the concept and all further ideas were born by joint efforts. Initially, we thought of doing a hackathon. But during the development of the concept, the idea turned into a quest, after all, a technical support engineer is a different kind of activity than programming.

So, we called friends, comrades, acquaintances, different people helped us with the concept - one person with T2 (the second line of support is Ed.), one person with T3, a couple of people from the SWAT team (fast response team for especially urgent cases - Ed.). We all got together, sat down and tried to come up with tasks for our quest.

— It was very unexpected to find out all about this, because, as far as I know, usually quest mechanics are worked out by scriptwriters, that is, not only did you deal with such a complex thing, but also in relation to your work, to your professional field of activity.

- Yes, we wanted to make not just entertainment, but to "pump" the technical skills of engineers. One of the tasks in our department is the exchange of knowledge and training, but such a quest is a great opportunity to let people “touch” some new techniques for them live.

How did you come up with assignments?

- Brainstormed. We had an understanding that we should make some technical tests, and such that they were interesting and at the same time carried new knowledge.
For example, we thought that people should be allowed to try sniffing traffic, use hex editors, do something for Linux, some slightly deeper things related to our products (Veeam Backup & Replication and others).

Also important was the concept. We decided to start from the theme of hackers, anonymous access and the atmosphere of secrecy. The Guy Fawkes mask was made a symbol, and the name came by itself - Veeamonymous.

"In the beginning was the word"

To stir up interest, we decided to organize a PR campaign on the topic of the quest before the start of the event: we hung posters with an announcement around our office. And a few days later, secretly from everyone, they themselves painted them with spray cans and launched a “duck”, they say, some attackers ruined the posters, even attached a photo with a proof ....

- So you did it yourself, that is, the organizing team ?!

- Yes, on Friday, at 9 o'clock, when everyone had already left, we went and drew the letter "V" in green from the balloons.) Many participants in the quest did not guess who did it - people came up to us and asked who ruined the posters ? Someone very seriously approached this issue and arranged a whole investigation on this topic.

For the quest, we also wrote audio files, “ripped out” sounds: for example, when an engineer logs into our [production CRM] system, there is an answering machine robot that says all sorts of phrases, numbers ... Here we are from those words that he has recorded, composed more or less meaningful phrases, well, maybe a little crooked - for example, we got "No friends to help you" in an audio file.

For example, we represented the IP address in binary code, everything, again, with the help of these numbers [pronounced by the robot], all sorts of frightening sounds were added. We filmed the video ourselves: on the video we have a man sitting in a black hood and wearing a Guy Fawkes mask, but in fact there is not one person, but three, because two are standing behind him and holding a “background” from a blanket :).

“Well, you’re confused, to be honest.

Yes, we are on fire. In general, at first they came up with our technical tasks, and then they composed a literary and gaming canvas on the topic of what supposedly happened. According to the scenario, the participants were hunting for a group of hackers called "Veeamonymous". The idea was also that we, as it were, “break the 4th wall”, that is, we transfer events into reality - here we drew from a spray can, for example.

With the literary processing of the text, one of the native English speakers from our department helped us.

“Wait, why do you need a native speaker?” Did you do it all in English too?!

— Yes, we held for the St. Petersburg and Bucharest offices, so everything was in English.

For the first experience, we tried to make everything just work, so the scenario was linear and quite simple. Added more entourage: secret texts, ciphers, pictures.

We also used memes: there were a lot of pictures on the topics of investigations, UFOs, some popular horror stories - some teams got distracted by this, tried to find some hidden messages there, apply their knowledge of steganography and other things ... but, of course, there is nothing like that was.

About thorns

However, in the process of preparation, we also encountered tasks that were unexpected for ourselves.

They fought a lot over them and solved all sorts of suddenly arising issues, and about a week before the quest they generally thought that everything was gone.

Probably, it is worth talking a little about the technical basis of the quest.

Everything was done on our internal ESXi lab. We had 6 teams, so we had to allocate 6 resource pools. So, for each team, we deployed a separate pool with the necessary virtual machines (same IPs). But since all this was on servers that lie on the same network, the current configuration of our VLANs did not allow isolating machines in different pools. And, for example, during a test run, we got situations where a machine from one pool connected to a machine from another.

How could you fix the situation?

- At first, we thought for a long time, testing all sorts of options with permissions, separate vLANs for machines. As a result, they did this - each team sees only the Veeam Backup server, through which all further work takes place, but does not see the hidden subplot in which they are:

• multiple Windows machines
• Windows core server
• linux machine
• pair of VTL (Virtual Tape Library)

All pools are assigned a separate group of ports on the vDS switch and their own Private VLAN. Such double isolation is just needed to completely exclude the possibility of network interaction.

About the brave

- Anyone could take part in the quest? How were the teams formed?

— It was our first experience of holding such an event, and the capacity of our laboratory was limited to 6 teams.

First, as I said, we conducted a PR campaign: using posters and mailing lists, we announced that a quest would be held. We even had some clues - phrases in binary code were encrypted on the posters themselves. In this way, we got people interested, and people themselves agreed among themselves with friends, with buddies, and cooperated. As a result, more applicants responded than we had pools, so we had to make a selection: we came up with a simple test task and sent it to everyone who responded. It was a logic puzzle, it had to be solved quickly.

The team was allowed up to 5 people. The captain was not required there, the idea was in cooperation, in communication with each other. Someone is strong, for example, in Linux, someone is strong in teips (backups to tapes), and everyone, seeing the task, could put their efforts into a common solution. Everyone communicated with each other, found a solution.

- And at what point did this event start? Did you have any "hour X"?

- Yes, we had a strictly appointed day, we chose it so that there was less workload in the department. Naturally, team leaders were informed in advance that such and such teams were invited to participate in the quest, and they needed to be given some relief [regarding loading] on that day. Everything seemed to indicate that it should be the end of the year, December 28, Friday. We expected to take about 5 hours, but all the teams did it faster.

- Everyone was on an equal footing, did everyone have the same tasks based on real cases?

- Well, yes, each of the compilers took some stories from personal experience. We knew about something that this could be in reality, and it would be interesting for a person to “feel” it, look, figure it out. They also took some more specific things - for example, data recovery from damaged tapes. Some with hints, but most teams managed on their own.

Or it was necessary to use the magic of quick scripts - for example, we had a story that some kind of “logic bomb” “torn” a multi-volume archive into random folders along the tree, and we had to collect data. You can do it manually - find and copy [files] one at a time, or you can write a script using a mask.

In general, we tried to adhere to the point of view that one problem can be solved in different ways. For example, if you are a little more experienced or want to "get confused", then you can solve it faster, and there is a direct way to solve it "on the forehead" - but at the same time you will spend more time on the task. That is, almost every task had several solutions, and it was interesting which paths the teams would choose. So the non-linearity was precisely in the choice of the solution.

By the way, the Linux problem turned out to be the most difficult - only one team solved it on its own, without prompting.

Could you take hints? Like in a real quest??

- Yes, it was possible to take, because we understood that people are different, and those who lack some kind of knowledge could get into the same team, so in order not to delay the passage and the competitive interest did not disappear, we decided that they would hints. To do this, each team was observed by a person from the organizers. Well, we made sure that no one cheated.

About the stars

- Were there any prizes for the winners?

— Yes, we tried to make the most pleasant prizes both for all participants and for the winners: the winners received designer sweatshirts with the Veeam logo and a phrase encrypted in a hexadecimal code in black). All participants received a Guy Fawkes mask and a branded bag with the logo and the same code.

- That is, everything was like in a real quest!

— Well, we wanted to do a cool, adult thing, and I think we succeeded.

- This is true! And what was the final reaction of those who participated in this quest? Have you achieved your goals?

- Yes, many came up later, saying that they clearly saw their weak points and wanted to tighten them up. Someone stopped being afraid of certain technologies - for example, dumping blocks from teips and trying to get something out there ... Someone realized that he needed to pull up Linux, and so on. We have tried to give a fairly wide range of problems, but not quite trivial.

Winning Team

"Whoever wants, he will achieve!"

- Did it require a lot of effort from those who prepared the quest?

- In fact yes. But this was most likely due to the fact that we had no experience in preparing such quests, such infrastructures. (Let's make a reservation that this is not our real infrastructure - it just had to perform some game functions.)

For us it was a very interesting experience. At first I was skeptical, because the idea seemed to me even too cool, I thought that it was very difficult to implement. But they started doing it, started to plow, everything started to catch fire, and in the end we succeeded. And there were even almost no overlays.

In total we spent 3 months. For the most part, we came up with a concept, discussed what we could implement. In the process, of course, something changed, because we understood that we didn’t have the technical ability for something to do it. On the go, I had to redo something, but so that the whole canvas, history and logic would not break. We tried not just to give a list of technical tasks, but to make it fit into the story, so that it was coherent and logical. The main work went on for the last month, that is, 3-4 weeks before X-day.

- That is, in addition to your main activity, did you allocate time for preparation?

- We did this in parallel with the main work, yes.

Are you being asked to do this again?

— Yes, we have many requests to repeat.

- And you?

- We have new ideas, new concepts, we want to attract more people and stretch it out in time - both the selection process and the game process itself. In general, we are inspired by the Cicada project, you can google it - this is a very cool IT topic, where people from all over the world unite, start branches on reddit, on forums, they use the translation of ciphers, and solve riddles, and all that.

- The idea was great, just respect for the idea and implementation, because it is really worth a lot. I sincerely wish you not to lose this inspiration, so that all your new projects are also successful. Thank you!

— Yes, but will it be possible to look at an example of a task that you definitely will not reuse?

“I suspect we won't reuse any of them. Therefore, I can tell about the course of the entire quest.

Bonus trackAt the very beginning, players have the name of the virtual machine and credentials from vCenter. Logged into it, they see this machine, but it does not start. Here you have to guess that something is wrong with the .vmx file. After downloading it, they see the hint needed for the second step. In fact, it says that the database used by Veeam Backup & Replication is encrypted.
After removing the prompt, uploading the .vmx file back and successfully turning on the machine, they see that one of the disks does indeed contain a base64-encrypted base. Accordingly, the task is to decrypt it and get a fully functional Veeam server.

A little about the virtual machine on which this all happens. As we remember, according to the plot, the protagonist of the quest is a rather dark person and is engaged in something that is clearly not very legal. Therefore, his work computer should have quite a hacker look that we had to create, despite the fact that it is Windows. First of all, a lot of props were added, such as information on major hacks, DDoS attacks, and the like. Then we installed all the typical software and spread different dumps, files with hashes, etc. everywhere. Everything is like in the movies. Among other things, there were folders named according to the principle closed-case *** and open-case ***
To progress further, players need to restore hints from backup files.

It must be said here that at the beginning the players were given quite a bit of information, and most of the data (like IP, logins and passwords) they receive during the quest, finding clues in backups or files scattered on machines. Initially, the backup files are located in the Linux repository, but the folder itself on the server is mounted with the flag noexec, so the agent responsible for file recovery cannot start.

Once the repository is repaired, members have access to all content and can finally restore any information. It remains to understand which one. And for this, they just need to study the files stored on this machine, determine which of them are “broken” and what exactly needs to be restored.

At this point, the scenario shifts away from general IT knowledge towards specific Veeam features.

In this particular example (when you know the file name but don't know where to look for it), you need to use the search function in Enterprise Manager, and so on. As a result, after restoring the entire logical chain, the players have one more login/password and nmap output. This brings them to a Windows Core server, moreover, via RDP (so that life does not seem like honey).

The main feature of this server: with the help of a simple script and several dictionaries, an absolutely meaningless structure of folders and files was formed there. And when you login, you get a welcome message like “A logic bomb exploded here, so you will have to piece together hints for further steps.”

The next hint was divided into a multi-volume archive (pieces 40-50) and randomly distributed into these folders. Our idea was that players should show their talents in writing simple PowerShell scripts in order to put together a multi-volume archive using a known mask and get the desired data. (But it turned out like in that joke - some of the subjects turned out to be unusually physically developed.)

The archive contained a photo of the cassette (with the inscription "Last Supper - Best Moments"), which hinted at the use of an attached tape library, where there was a cassette with a similar name. Here's just one trouble - it turned out to be so inoperable that it was not even catalogued. Here began, probably, the most hardcore part of the quest. We erased the header from the cassette, so in order to restore data from it, you just need to dump the “raw” (raw) blocks and view them in a hex editor to find file start markers.
We find the marker, look at the offset, multiply the block by its size, add the offset, and using the internal tool, we try to restore the file from a certain block. If everything is done correctly and the math agrees, then the players have a .wav file in their hands.

In it, with the help of a voice generator, among other things, a binary code is dictated, which is revealed in another IP.

This, it turns out, is a new Windows server, where everything hints at the need to use Wireshark, only it is not there. The main trick is that two systems are installed on this machine - only the disk from the second is disabled offline through the device manager, and the logical chain leads to the need to reboot. After that, it turns out that by default a completely different system should boot, where Wireshark is installed. And all this time we were on the secondary OS.

There is nothing special to do here, it is enough to enable capture on a single interface. A relatively close examination of the dump reveals a clearly left packet sent from an auxiliary machine at regular intervals, which contains a link to a youtube video where players are asked to call a specific number. The first caller will listen to the congratulations on the first place, the rest - an invitation to HR (just kidding)).

By the way, we are open Vacancies for technical support engineers and for trainees. Welcome to the team!

Source: habr.com