This winter, specifically on one of the days between Christmas and New Year's, the Veeam technical support engineers were busy with quite unusual tasks: they were hunting for a group of hackers called 'Veeamonymous'.
About how the team came up with and conducted a real-life quest at work, with tasks 'close to combat', told Kirill Stetsko, Escalation Engineer.
β Why did you even start this?
β Just like people invented Linux at one time β just for fun, for their own enjoyment.
We wanted some excitement, and at the same time, we wanted to do something useful, something interesting. Plus, it was necessary to provide some emotional relief for the engineers from their daily routine.
β Who suggested this? Whose idea was it?
β The idea came from our manager Katya Egorova, and then the concept and all further ideas were born from our joint efforts. Initially, we thought about organizing a hackathon. However, during the concept development, the idea evolved into a quest, after all, technical support engineers have a different kind of activity compared to programming.
So, we called friends and acquaintances, different people helped us with the concept β one person from T2 (second-line support β editor's note), one person from T3, a couple of people from the SWAT team (the quick reaction team for particularly urgent cases β editor's note). We all gathered together, sat down, and tried to come up with tasks for our quest.
β It was very unexpected to learn about all this, because, as far as I know, usually quest mechanics are developed by scenario specialists, which means that not only were you dealing with such a complex task, but also in relation to your work, to your professional field.
β Yes, we wanted to create not just entertainment, but to 'pump' the technical skills of the engineers. One of the tasks in our department is knowledge exchange and training, and this quest is a great opportunity to let people 'touch' some new techniques live.
β How did you come up with the tasks?
β We organized a brainstorming session. We had an understanding that we had to create certain technical challenges, and they had to be interesting while simultaneously conveying new knowledge.
For example, we thought it would be nice to let people try sniffing traffic, using hex editors, doing some deeper tasks related to our products (Veeam Backup & Replication and others).
Another important part was the concept. We decided to base it on the theme of hackers, anonymous access, and an atmosphere of secrecy. We made the Guy Fawkes mask our symbol, and the name came naturally β Veeamonymous.
"In the beginning was the word"
To spark interest, we decided to launch a PR campaign themed around a quest before the event: we hung posters with the announcement around our office. A few days later, secretly from everyone, we spray-painted them and started a rumor that some vandals had damaged the posters, even attaching a photo as evidence....
β So you did it yourself, meaning the organizing team?!
β Yes, on Friday around 9 PM, when everyone had already left, we went and spray-painted a green 'V'. Many quest participants never figured out who did it β people came up to us asking who had ruined the posters. Someone took this question very seriously and even conducted a whole investigation into it.
For the quest, we recorded audio files, 'extracting' sounds: for example, when an engineer logs into our [production CRM] system, there is a robot auto-responder that says various phrases, numbersβ¦ From those recorded words, we formed more or less meaningful phrases, albeit a bit awkward β for instance, we came up with "No friends to help you" in the audio file.
An IP address, for example, we represented in binary code, again using those numbers [pronounced by the robot], adding various scary sounds. We filmed a video: in the video, a person sits in a black hoodie and a Guy Fawkes mask, but in reality, there isnβt just one person; there are three, because two are behind him holding a 'background' made of a blanket:).
β You really went all out, to put it mildly.
β Yes, we got fired up. In general, we first came up with our technical assignments and then created a literary and game framework about what supposedly happened. According to the scenario, the participants were hunting a group of hackers called 'Veeamonymous'. The idea was also that we were kind of 'breaking the fourth wall', that is, bringing the events into reality β for example, we painted using spray paint.
One of the native English speakers from our department helped us with the literary processing of the text.
β Wait, why a native speaker? You did all this in English too?!
β Yes, we held it for the offices in St. Petersburg and Bucharest, so everything was in English.
For our first experience, we tried to make everything simply work, so the scenario was linear and quite simple. We added more atmosphere: secret texts, ciphers, images.
We also used memes: there were a bunch of pictures on topics like investigations, UFOs, and some popular horror stories β some teams got distracted by this and tried to find hidden messages, applying their knowledge of steganography and the likeβ¦ but of course, there was nothing like that.
About the thorns
However, during the preparation process, we faced unexpected tasks ourselves.
We struggled with them a lot and solved various suddenly arising issues, and about a week before the quest, we even thought that it was all lost.
It might be worth telling a bit about the technical foundation of the quest.
Everything was done on our internal ESXi lab. We had 6 teams, so we needed to allocate 6 resource pools. Thus, for each team, we deployed a separate pool with the necessary virtual machines (same IPs). But since all this was on servers that were in one network, the current configuration of our VLANs did not allow us to isolate machines in different pools. And, for instance, during the test run, we encountered situations where a machine from one pool connected to a machine from another.
β How were you able to fix the situation?
β At first, we thought for a long time, testing various options with permissions, separate VLANs for machines. In the end, we did it this way β each team only sees the Veeam Backup server, through which all further work is done, but does not see the hidden subpool, which contains:
- several Windows machines
- Windows core server
- machine with Linux
- pair of VTL (Virtual Tape Library)
All pools were assigned a separate group of ports on the vDS switch and their own Private VLAN. Such double isolation is necessary to completely eliminate the possibility of network interaction.
About the brave ones
β Could anyone interested participate in the quest? How were the teams formed?
β It was our first experience conducting such an event, and our lab's capacity was limited to 6 teams.
Initially, as I mentioned, we conducted a PR campaign: using posters and mailings, we informed that a quest would take place. We even had some hints β the posters contained phrases encoded in binary code. This way, we captured people's interest, and they coordinated among themselves with friends, forming teams. In the end, more people responded than we had pools available, so we had to conduct a selection: we came up with a simple test task and sent it to everyone who replied. It was a logical puzzle that needed to be solved quickly.
A team could have up to 5 people. A captain was not required; the idea was cooperation and communication among team members. Some were strong in Linux, others were strong in tapes (backups on tapes), and everyone, seeing the task, could contribute their efforts to the common solution. Everyone communicated, found solutions together.
β When did this event start? Did you have a specific 'hour X'?
β Yes, we had a strictly designated day; we chose it to minimize the load in the department. Naturally, we informed the team leads in advance that certain teams were invited to participate in the quest and needed some leniency [regarding workload] on that day. It turned out that it had to be at the end of the year, December 28, a Friday. We expected it to take about 5 hours, but all teams completed it faster.
β Everyone was on equal footing, and all had the same tasks based on real cases?
Well, yes, each of the creators drew from their personal experiences some stories. For some things, we knew they could happen in reality and it would be interesting for a person to 'feel' it, look at it, understand it. We also took some more specific things β for example, data recovery from damaged tapes. Some used hints, but most teams managed on their own.
Or we had to apply the magic of quick scripts β for instance, we had a story where a 'logical bomb' 'exploded' a multi-volume archive into random folders in a tree, and we needed to gather the data. You could do this manually β finding and copying [files] one by one β or you could write a script based on a mask.
In general, we tried to adhere to the viewpoint that one task can be solved in different ways. For example, if you're a little more experienced or want to 'overthink' it, you can solve it faster, while there is a straightforward 'brute force' way β but you will spend more time on the task. That is, almost every task had several solutions, and it was interesting to see which paths the teams would choose. So, the non-linearity was in the choice of solution.
By the way, the hardest task was the Linux one β only one team solved it independently, without hints.
Could you take hints? Like in a real quest?
Yes, hints could be taken because we understood that people are different, and those with less knowledge might end up in the same team. So to not prolong the gameplay and keep the competitive interest alive, we decided that hints would be available. For this, a person from the organizing team observed each team. And we also made sure that no one cheated.
About the stars
Were there prizes for the winners?
Yes, we tried to make the prizes as enjoyable as possible for all participants as well as for the winners: the winners received designer hoodies with the Veeam logo and a phrase encoded in hexadecimal, in black). All participants received a Guy Fawkes mask and a branded bag with the logo and the same code.
So everything was just like in a real quest for you!
Well, we wanted to create something cool and adult, and I think we succeeded.
β That's right! What was the reaction from those who participated in this quest? Did you achieve your goals?
β Yes, many approached us afterward, saying they clearly saw their weak points and wanted to improve them. Some overcame their fear of certain technologies β for example, extracting blocks from tapes and trying to figure something outβ¦ Others realized that they need to strengthen their Linux skills, and so on. We tried to provide a sufficiently broad range of tasks that weren't entirely trivial.
Winning team
βWho wants it, achieves it!β
β Did it require a lot of effort from those who prepared the quest?
β Overall, yes. But that was likely due to the fact that we had no experience in preparing such quests or that kind of infrastructure. (Letβs clarify that this is not our real infrastructure β it was meant to perform certain game functions.)
It was a very interesting experience for us. Initially, I was skeptical because the idea seemed too ambitious; I thought it would be very difficult to implement. But as we began to work on it, we got motivated, and in the end, we succeeded. And it went very smoothly without major issues.
In total, we spent 3 months on it. For the most part, we were coming up with the concept and discussing what we could implement. Naturally, some things changed during the process because we realized there were technical limitations for certain aspects. We had to adjust on the fly but ensured that the overall storyline, history, and logic remained intact. We weren't just trying to provide a list of technical tasks, but rather weave them into a story that was coherent and logical. The main work happened in the last month, that is, 3-4 weeks before the big day.
β So, apart from your main activities, you allocated time for preparation?
β We did this alongside our primary work, yes.
β Are you being asked to conduct something like this again?
β Yes, we have many requests to repeat it.
β And what about you?
We have new ideas, new concepts; we want to attract more people and spread this out over time β both the selection process and the game itself. In general, we are inspired by the 'Cicada' project, which you can google β itβs a really cool IT theme where people from all over the world come together, starting threads on Reddit and forums, using code translations, solving puzzles, and so on.
The idea was excellent; I genuinely respect the idea and its implementation because it's truly valuable. I sincerely wish that you donβt lose this inspiration and that all your new projects are equally successful. Thank you!
Can we see an example of a task that you definitely wonβt reuse?
I suspect we wonβt reuse any of them. Therefore, I can tell you about the entire quest process.
Bonus TrackAt the very beginning, players have the name of the virtual machine and credentials for vCenter. Once logged in, they see the machine, but it doesn't start. Here, they need to figure out that something is wrong with the .vmx file. After downloading it, they see a hint necessary for the second step. Essentially, it states that the database used by Veeam Backup & Replication is encrypted.
By removing the hint, re-uploading the .vmx file, and successfully starting the machine, they see that one of the disks indeed contains a database encrypted in base64. Consequently, the task is to decrypt it and obtain a fully functional Veeam server.
A little about the virtual machine where all this is happening. As we remember, according to the plot, the main character of the quest is a rather dark personality involved in something clearly illegal. Therefore, his working computer should have a distinctly hacker-like appearance, which we had to create despite it being Windows. First, we added a lot of props, like information on major hacks, DDoS attacks, and similar topics. Then we installed various typical software and scattered different dumps and hash files everywhere. Just like in the movies. Among other things, there were folders named in the style of closed-case*** and open-case***.
To progress further, players need to restore hints from files in the backups.
It's worth noting that at the beginning, players were given very little information, and most data (like IPs, logins, and passwords) is obtained throughout the quest by finding clues in backups or files scattered across machines. Initially, backup files are located on a Linux repository, but the folder itself on server is mounted with the flag noexec, so the agent responsible for recovering the files cannot start.
After fixing the repository, participants gain access to all content and can finally restore any information necessary. They just need to figure out which specific information they need. For this, they must examine the files stored on this machine, determine which ones are 'corrupted,' and identify exactly what needs to be recovered.
At this stage, the scenario shifts from general IT knowledge to specific functions of Veeam.
In this particular example (when you know the file's name but not where to find it), you need to use the search function in Enterprise Manager, and so on. Eventually, after restoring the entire logical chain, players have another login/password and the output from nmap. This leads them to a Windows Core server, using RDP (to add a bit of challenge).
The main highlight of this server is that, using a simple script and several dictionaries, an absolutely nonsensical structure of folders and files was created. Upon logging in, a welcome message appears stating, 'A logical bomb exploded here, so you'll need to gather clues for the next steps piece by piece.'
The next clue was split into a multi-part archive (about 40-50 pieces) and randomly distributed across these folders. Our intention was for players to demonstrate their skills in writing simple PowerShell scripts in order to compile the multi-part archive using a known pattern and obtain the sought-after data. (However, it turned out like in that joke β some of the subjects were surprisingly physically developed.)
The archive contained a photo of a cassette (labeled 'Last Supper - Best Moments'), hinting at the use of a connected tape library, where there was a cassette with a similar name. The only problem was that it turned out to be so inoperable that it wasn't even cataloged. This seemed to mark the beginning of the most hardcore part of the quest. We erased the title of the cassette, so to recover the data from it, we just needed to dump the raw blocks and review them in a hex editor to find the file start markers.
We find the marker, check the offset, multiply the block by its size, add the offset and, using an internal tool, we attempt to recover the file from the specific block. If everything is done correctly and the math checks out, the players end up with a .wav file.
In it, using a voice generator, a binary code is dictated which reveals yet another IP.
It turns out to be a new Windows server, where everything hints at the need for Wireshark, but it isn't installed there. The main focus is that this machine has two systems installed β only the disk with the second one is disabled via the device manager while offline, and the logical chain leads to the need for a reboot. After that, it turns out that a completely different system, where Wireshark is installed, should load by default. And all this time, we were on the secondary OS.
Here, not much needs to be done, just start the capture on the only interface. With relatively careful examination of the dump, a clearly suspicious packet is seen, sent from the auxiliary machine at regular intervals, which contains a link to a YouTube video where players are asked to call a specific number. The first caller will hear a congratulation for coming in first place, while the others will receive an invitation to HR (just a joke)).
By the way, we have open for technical support engineers and interns. Welcome to the team!
Source: habr.com
