On March 25, we opened registration for , a game for fans and experts of the Linux operating system. Some statistics: 1,117 people registered for the game, of which 317 found at least one key, 241 successfully completed the first stage task, 123 completed the second, and 70 passed the third stage. Today our game has come to an end, and we congratulate our winners!
- The first place was taken by Alexander Teldekov.
Alexander described himself as a typical system administrator. He lives in Volgograd and has been administering various Unix-like systems for about twenty years. He has worked at Internet service providers, a bank, and a system integrator. Currently, he works remotely at a small company, focusing on cloud infrastructure for a major foreign client. He enjoys reading and listening to music. Regarding the game, Alexander said he generally liked it and enjoys such challenges. He encountered something similar to Hackerrank during an interview at one of the companies, which was interesting. - Second place — Roman Suslov.
Roman is from Moscow. He is 37 years old. He works as a Linux/Unix engineer at 'Infosystems Jet.' His job involves administering and troubleshooting Linux/Unix systems + SAN. He has diverse interests: Linux systems, programming, reverse engineering, information security, Arduino. About the game, Roman noted that he generally enjoyed it. "I stretched my brain a bit and took a break from the gray everyday work routine. 🙂 I would have liked more tasks because just as I started to get into it, the game was already over." - Third place — alex3d.
Alex lives in Moscow and works in software development. "Thanks for the contest, it was interesting to test my google-fu skill."
Also in the ranking of the top 10 players:
- Yevgeniy Saldayev
- Markel Mokhnachevskiy
- Konstantin Konosov
- Pavel Sergeev
- Vladimir Bovaev
- Ivan Bubnov
- Pavlo Klets
We understand that there are many ways to solve all our tasks; below are some possible solutions described.
1. First stage
We called it 'Are you really an admin?' since the task was quite simple — to fix a warm, cozy service.
1.1. Interesting facts:
Two players found the first key within the first 15 minutes of the game, and within the first hour, we had three leaders who completed the task.
1.2. Task
You started working at a company where there hasn't been a qualified IT specialist for a long time. Before you can restore order, you need to resolve a pressing issue that is blocking the office's operations.
The cleaner accidentally knocked the power cable of the server cabinet with a mop. Power has been restored, but a very important website still isn’t working. The website is crucial because the company isn't very concerned with information security, and the administrator password for the CEO's computer can be found openly on the homepage.
Recently, the password was changed, and everyone forgot the new one, which is preventing the director from working. Rumors are circulating that there were also keys on this machine that could help us decrypt the backup of the accounting documents.
Everyone is waiting for a quick resolution of the issue!
1.3. Solution
1. First, we need to change the root password on the virtual machine to gain access to it. Upon startup, we notice that this is Ubuntu 16.04 Server.
To reset the root password, we restart the machine. During boot, when the grub menu appears, we switch to editing the Ubuntu entry by pressing the 'e' key. We edit the linux line, adding at the end init=/bin/bash. We load using Ctrl+x, getting to the bash. We remount the root with rw and change the password:
$ mount -o remount,rw /dev/mapper/ubuntu--vg-root
$ passwdDon't forget about sync, then reboot.
2. The condition states that our web server is not working; let's check:
$ curl localhost
Not Found
The requested URL / was not found on this server.
Apache/2.4.18 So, in fact, Apache is running, but it responds with a 404 code. Let's check the config:
$ vim /etc/apache2/sites-enabled/000-default.confHere we find the key — StevenPaulSteveJobs.
Let's check the path /usr/share/WordPress — it doesn't exist, but there is /usr/share/wordpress. We edit the config and restart Apache.
$ systemctl restart apache23. Let's try again; we get an error:
Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/share/wordpress/wp-includes/wp-db.php on line 1488Is the database not running?
$ systemctl status mysql
Active: active (running) What's the issue then? We need to investigate. To do this, we need to access MySQL, as described in . One of the documentation points recommends that we specify the option skip-grant-tables downward API support (simultaneously with this in /etc/mysql/mysql.conf.d/mysqld.cnf. Here too, we find the key — AugustaAdaKingByron.
We adjust the permissions for the user 'wp'@'localhost'. We start MySQL, making it accessible over the network by commenting out the option in the config skip-networking.
4. After the steps taken, the web server starts, but the site still isn't working, as
Warning: require_once(/usr/share/wordpress/wp-content/themes/twentysixteen/footer.php): failed to open stream: Permission denied in /usr/share/wordpress/wp-includes/template.php on line 562Let's change the file permissions.
$ chmod 644 /usr/share/wordpress/wp-content/themes/twentysixteen/footer.php
We refresh the page, go to the site, and find the key — BjarneStroustrup! We have found all three keys, our director can work, we decrypted the accounting files. Everyone is happy, but you have a lot of work ahead in setting up infrastructure, backups, and security in the company.
2. Second Stage
We needed to solve the task of gathering analytics. Everyone loves analytics — who is coming from where and in what quantities. We devised a case that every engineer may encounter in one form or another in their life.
2.1. Interesting Facts
One of our players entered the correct key within the first 10 minutes of the game, and within the first hour, we had a leader who completed the task.
2.2. Task
You started working at the company, managers came to you and asked to find out who the emails from Africa were sent to. You need to create a top-21 list of recipient addresses from them. The first letters of the recipient addresses are the key. One caveat: the mail server that the emails were sent through won’t boot. Everyone is waiting for an operational solution to this issue!
2.3. Solution
1. The server won’t boot due to a non-existent swap partition in fstab; during boot, the system tries to mount it and crashes. How to boot?
We download the image, we downloaded CentOS 7, boot from Live CD/DVD (Troubleshooting -> Rescue), mount the system, make changes. /etc/fstabRight here we find the first key — GottfriedWilhelm11646Leibniz!
Create swap:
$ lvcreate -n swap centos -L 256M
$ sync && reboot2. As always, there is no password, we need to change the root password on the virtual machine. We have done this in the first task. We change it and successfully log into the server, but it immediately reboots. The server is rebooting so fast that we can’t even carefully look at all the logs. How to understand what is happening?
Again, we boot from livecd, carefully examine the system logs, and as a precaution, check the cron since it has such periodicity. There we find the problem and the second key — Alan1912MathisonTuring!
We need to /etc/crontab delete or comment out the line echo b > /proc/sysrq-trigger.
3. After the server has booted, you can carry out the task set by the managers: 'What are the IP addresses of Africa?' This information is generally publicly available. You can find it online using phrases like 'ip address africa', 'geoip database'. To solve this task, you can use freely available IP distribution databases (geoip). We used a database as a reference. , available under the Creative Commons Attribution-ShareAlike 4.0 license.
Let's try to solve our task using only Linux system utilities, but it can actually be addressed in a multitude of ways: through text filtering utilities and scripts in various programming languages.
To start, we'll simply obtain pairs of 'sender IP — recipient' from the mail log /var/log/maillog (we'll construct a table of email recipients — sender IP). This can be done with the following command:
$ cat /var/log/maillog | fgrep -e ' connect from' -e 'status=sent' | sed 's/[][]/ /g' | awk '/connect from/ {ip=$11} /status=sent/ {print $10" "ip}' > log1.txtAnd before we continue with compiling the database of African addresses, let's take a look at the top sender IP addresses.
$ cat log1.txt | cut -d' ' -f1 | sort | uniq -c | sort -r | head -n 40
5206 L2JhbjAbM67GA99jg@mail.ru
4165 iHKTBkegOQa6fIALq@mail.ru
3739 nHkcBl7BdgXxijSYD7@mail.ru
3405 SMAzPJAzbl9vp4hAXo@mail.ru
3346 xILz6d7P@mail.ru
Among all, the first three recipients in the top stand out significantly in terms of the number of emails received. If we grep the sender IP addresses that sent to the addresses from this top-3, we can notice a clear predominance of certain networks:
$ cat log1.txt | fgrep 'L2JhbjAbM67GA99jg@mail.ru' | cut -d' ' -f2 | sort | cut -d'.' -f1 | uniq -c | sort -r | head
831 105
806 41
782 197
664 196
542 154
503 102
266 156
165 45
150 160
108 165The majority of the networks 105/8, 41/8, 196/8, 197/8 are allocated by AFRINIC — one of the five regional internet registrars responsible for the distribution of internet resources. AFRINIC distributes the address space in Africa. Moreover, 41/8 fully belongs to AFRINIC.
https://www.nic.ru/whois/?searchWord=105.0.0.0
https://www.nic.ru/whois/?searchWord=41.0.0.0
Thus, the answer to the task can actually be found in the log itself.
$ cat log1.txt | fgrep -e '105.' -e '41.' -e '196.' -e '197.' -e '154.' -e '102.' | awk '{print $1}' | sort | uniq -c | sort -r | head -n 21
4209 L2JhbjAbM67GA99jg@mail.ru
3313 iHKTBkegOQa6fIALq@mail.ru
2704 nHkcBl7BdgXxijSYD7@mail.ru
2215 uvRbp1O@mail.ru
1774 sPmMsmmFiV@mail.ru
1448 BtG3aHgQgCKuze2AKuRH@mail.ru
1233 eQpuuQ2uQdbwRL3@mail.ru
958 nJT5dpaBZ@mail.ru
862 ef4WbQiB@mail.ru
762 dQCqKL6eVminFfH7wLA@mail.ru
632 ifq6Rd1HxuCQOdO9@mail.ru
539 cFwm2ssypMmx1sA7@mail.ru
531 twtTnr4G@mail.ru
431 TSrczgYASrR11Hs3qCi@mail.ru
380 o3r3exc3OL@mail.ru
357 rzmjr2VAHK@mail.ru
348 vnPr6YjJ3ndw@mail.ru
312 anOjFXrwOtLP2Rl1Vcz6@mail.ru
289 dvny5zHmRW8fiT@mail.ru
282 sgg9jPxFDYvzw8Kr@mail.ru
274 tKSevzA7GntJ@mail.ruAt this stage, we obtain the string 'LinuxBenedictTorvadst'.
Correct key: 'LinusBenedictTorvalds'.
The obtained string contains a typo compared to the correct key in the last 3 characters. This is due to the fact that the networks we chose are not fully allocated to the countries of Africa and how emails are distributed across IP addresses in our log.
With sufficient clarification of the largest networks allocated to the countries of Africa, an accurate answer can be obtained.:
$ cat log1.txt | fgrep -e '105.{30..255}. -e '41. -e '196.{64..47}. -e '196.{248..132}. -e '197.{160..31}. -e '154.{127..255}. -e '102.{70..255}. -e '156.{155..255}. | awk '{print $1}' | sort | uniq -c | sort -r | head -n 21
3350 L2JhbjAbM67GA99jg@mail.ru
2662 iHKTBkegOQa6fIALq@mail.ru
2105 nHkcBl7BdgXxijSYD7@mail.ru
1724 uvRbp1O@mail.ru
1376 sPmMsmmFiV@mail.ru
1092 BtG3aHgQgCKuze2AKuRH@mail.ru
849 eQpuuQ2uQdbwRL3@mail.ru
712 nJT5dpaBZ@mail.ru
584 ef4WbQiB@mail.ru
463 dQCqKL6eVminFfH7wLA@mail.ru
365 ifq6Rd1HxuCQOdO9@mail.ru
269 cFwm2ssypMmx1sA7@mail.ru
225 twtTnr4G@mail.ru
168 TSrczgYASrR11Hs3qCi@mail.ru
142 o3r3exc3OL@mail.ru
111 rzmjr2VAHK@mail.ru
96 vnPr6YjJ3ndw@mail.ru
78 anOjFXrwOtLP2Rl1Vcz6@mail.ru
56 lHzWiB7ExvRtSbAcU9@mail.ru
56 dvny5zHmRW8fiT@mail.ru
40 sgg9jPxFDYvzw8Kr@mail.ru The task can also be solved in another way.
We download MaxMind, unpack it, and the following three commands also solve our task.
$ cat GeoLite2-Country-Locations-ru.csv | grep "Africa" | cut -d',' -f1 > africaIds.txt
$ grep -Ff africaIds.txt GeoLite2-Country-Blocks-IPv4.csv | cut -d',' -f1 > africaNetworks.txt
$ grepcidr -f africaNetworks.txt log1.txt | cut -d' ' -f1 | sort | uniq -c | sort -r | head -n21
In one way or another, we ultimately calculated the statistics, and the managers received the necessary data for their work!
3. Third Stage
The third stage is somewhat similar to the first — we also need to fix the warm lamp service, but it is more complex than in the first task.
3.1. Interesting Facts
In the first 15 minutes, three players found the first key, and 2 hours and 20 minutes after the start of the stage, our winner completed the task.
3.2. Assignment
You started working for a company where all company documents are stored on an internal Wiki server. Last year, an engineer ordered three new disks for the server in addition to one existing one, arguing that for system redundancy, disks need to be set up in certain arrays. Unfortunately, a few weeks after their installation, the engineer went on vacation to India and did not return.
For several years, the server operated without failures, but a couple of days ago, the company's network was hacked. According to instructions, the security staff extracted the disks from the server and sent them to you. During transport, one disk was irretrievably lost.
It is necessary to restore the functionality of the Wiki, primarily concerning the content of the wiki pages. A certain piece of text that was on one of the pages of this wiki is the password for the 1C server, which is urgently needed for its unlocking.
Additionally, somewhere on the wiki pages or elsewhere, there were passwords for the log server and the video surveillance server, which also need to be restored, as without them, the investigation of the incident is impossible. As always, we expect a prompt resolution to this issue!
3.3. Solution
1. We attempt to boot in sequence from the disks we have and receive the same message everywhere:
No bootable medium found! System halted
We need to boot from something. Once again, we are saved by booting from Live CD/DVD (Troubleshooting -> Rescue). Upon booting, we try to find the boot partition, do not find it, and end up in the shell. We try to study what and how with the disks. It is known that there are three. There are more tools for this in the 7th version of CentOS, where there are commands blkid or lsblk, which show us all the information about the disks.
What we do:
$ ls /dev/sd*It's immediately clear that
/dev/sdb1 - ext4
/dev/sdb2 - часть lvm
/dev/sda1 и /dev/sdc1 - части рейда
/dev/sda2 и /dev/sdc2 - про них ничего не известно на текущий моментWe mount sdb1, and it is clear that this is the boot partition of CentOS 6.
$ mkdir /mnt/sdb1 && mount /dev/sdb1 /mnt/sdb1Obviously, we go to the grub partition and find the first key—James191955Gosling in an unusual file.
2. We examine pvs and lvs since we are working with LVM. We see that there should be 2 physical volumes, one of which is not found and complains about a lost uid. We see that there should be 2 logical volumes: root and swap, with the root partially lost (attribute P for the volume). It cannot be mounted, which is a shame! We really need it.
There are also 2 more disks, we look at them, assemble and mount:
$ mdadm --examine --verbose --scan
$ mdadm --assemble --verbose --scan
$ mkdir /mnt/md127 && mount /dev/md127 /mnt/md127 We see that this is the boot partition of CentOS 6 and a duplicate of what is already on /dev/sdb1, and here again is the same key—DennisBMacAlistairCRitchie!
We look at how it is assembled /dev/md127.
$ mdadm --detail /dev/md127
We see that it should have been assembled from 4 disks, but was assembled from two /dev/sda1 and /dev/sdc1, which should have been numbers 2 and 4 in the system. We assume that it should also be possible to assemble the array from /dev/sda2 and /dev/sdc2 . It is unclear why there is no metadata on them, but that is the responsibility of the admin who is somewhere in Goa. We assume there should be RAID10 here, although there are options. We assemble:
$ mdadm --create --verbose /dev/md0 --assume-clean --level=10 --raid-devices=4 missing /dev/sda2 missing /dev/sdc2
We check blkid, pvs, lvs. We discover that we have assembled the physical volume that we were previously missing.
The lvroot has been repaired, we are mounting it, but first we activate VG:
$ vgchange -a y
$ mkdir /mnt/lvroot && mount /dev/mapper/vg_c6m1-lv_root /mnt/lvroot
And everything is there, including the key in the root home directory — /root/sweet.
3. We are trying to revive our server so it can boot normally. We move all logical volumes from our /dev/md0 (where we found everything) to /dev/sdb2, where the entire server originally operated.
$ pvmove /dev/md0 /dev/sdb2
$ vgreduce vg_c6m1 /dev/md0We power down the server, remove disks 1 and 3, leave the second, and boot from the Live CD/DVD in Rescue. We locate the boot partition and restore the bootloader in grub:
root (hd0,0)
setup (hd0)
We remove the boot disk and successfully boot, but the site does not work.
4. There are two options to start the site: set up Apache from scratch or use a pre-configured nginx with php-fpm:
$ /etc/init.d/nginx start
$ /etc/init.d/php-fpm start
Finally, we need to start MySQL:
$ /etc/init.d/mysqld start It does not start, and the clue lies in /var/log/mysql. Once you resolve the issue with MySQL, the site will work, and on the main page, there will be the key — RichardGCCMatthewGNUStallman! Now we have access to 1C, and employees will be able to receive their salaries. And you still have a lot of work ahead in setting up the infrastructure and security in the company.
We can also share once again the list of books that helped us and our participants prepare for the game: .
Thank you for being with us! Stay tuned for announcements of upcoming games!
Source: habr.com
