Once, in an interview, I was asked what would you do if you find a broken service due to the fact that the disk has run out of space?
Of course, I replied that I would see what this place was doing and, if possible, I would clean the place.
Then the interviewer asked, what if there is no free space on the partition, but you also don’t see the files that would take up all the space?
To this, I said that you can always look at open file descriptors, for example, with the lsof command and understand which application has taken all the available space, and then you can act according to the circumstances, depending on whether the data is needed.
The interviewer interrupted me at the last word, adding to his question: “Suppose we don’t need the data, it’s just a debug log, but the application is down because it can’t write the debug”?
“ok,” I replied, “we can turn off debugging in the application config and restart it.”
The interviewer objected: “No, we cannot restart the application, we still have important data in memory, and important clients are connected to the service itself, which we cannot force to reconnect.”
“okay,” I said, “if we can’t restart the application and we don’t care about the data, then we can just clear this open file via the file descriptor, even if we don’t see it in the ls command on the file system.”
The interviewer was satisfied, but I was not.
Then I thought, why doesn't the person testing my knowledge dig deeper? But what if the data is important after all? What if we can't restart the process, and at the same time this process writes to the file system on a partition that has no free space? What if we cannot lose not only the data already written, but also the data that this process is writing or trying to write?
Tuzik
At the beginning of my career, I was trying to create a small application that needed to store information about users. And then I thought, how can I match the user to his data. For example, I have Ivanov Ivan Ivanovich, and he has some data, but how to make friends with them? I can point out directly that the dog named "Tuzik" belongs to this same Ivan. But what if he changes his name and instead of Ivan becomes, for example, Olya? Then it will turn out that our Olya Ivanovna Ivanova will no longer have a dog, and our Tuzik will still belong to the non-existent Ivan. The database helped solve this problem, which gave each user a unique identifier (ID), and my Tuzik was tied to this ID, which, in fact, was just a serial number. Thus, the owner of the tuzik was with ID number 2, and at some point in time Ivan was under this ID, and then Olya became under the same ID. The problem of mankind and animal husbandry was practically solved.
File descriptor
The problem of a file and a program that works with this file is about the same as our dog and human. Suppose I opened a file named ivan.txt and started writing the word tuzik into it, but managed to write only the first letter “t” into the file, and this file was renamed by someone, for example, to olya.txt. But the file is the same and I still want to write my ace to it. Every time you open a file with a system call in any programming language, I get a unique ID that points me to a file, this ID is the file descriptor. And it doesn’t matter at all what and who does next with this file, it can be deleted, it can be renamed, it can change its owner or take away the rights to read and write, I will still have access to it, because at the time of opening the file I had the rights to read and / or write it and I managed to start working with it, which means I must continue to do so.
On Linux, the libc library opens 3 descriptor files for each running application (process), with numbers 0,1,2. More information you can find on the links и
- File descriptor 0 is called STDIN and is associated with application input.
- File descriptor 1 is called STDOUT and is used by output applications such as print commands.
- File descriptor 2 is named STDERR and is used by applications to output error messages.
If in your program you open any file for reading or writing, then most likely you will get the first free ID and this will be number 3.
You can see the list of file descriptors for any process if you know its PID.
For example, let's open a console with bash and see the PID of our process
[user@localhost ]$ echo $$
15771
In the second console, run
[user@localhost ]$ ls -lah /proc/15771/fd/
total 0
dr-x------ 2 user user 0 Oct 7 15:42 .
dr-xr-xr-x 9 user user 0 Oct 7 15:42 ..
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
You can safely ignore the file descriptor with number 255 within the framework of this article, it was opened for your needs by bash itself, and not by the linked library.
Now all 3 descriptor files are associated with the pseudo-terminal device , but we can still manipulate them, for example, run in the second console
[user@localhost ]$ echo "hello world" > /proc/15771/fd/0
And in the first console we will see
[user@localhost ]$ hello world
Redirect and Pipe
You can easily override these 3 descriptor files in any process, including in bash, for example, through a pipe (pipe) connecting two processes, see
[user@localhost ]$ cat /dev/zero | sleep 10000
You can run this command yourself with strace -f and see what's going on inside, but I'll make it short.
Our parent bash process with PID 15771 parses our command and understands exactly how many commands we want to run, in our case there are two of them: cat and sleep. Bash knows that it needs to create two child processes, and merge them into one pipe. In total, bash will need 2 child processes and one pipe.
Before creating child processes, bash runs a system call and receives new file descriptors on a temporary pipe buffer, but this buffer does not yet connect our two child processes in any way.
For the parent process, it looks like the pipe is already there, but there are no child processes yet:
PID command
15771 bash
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
Then using the system call bash creates two child processes, and our three processes will look like this:
PID command
15771 bash
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
PID command
9004 bash
lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 255 -> /dev/pts/21
PID command
9005 bash
lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 255 -> /dev/pts/21
Do not forget that clone clones the process along with all file descriptors, so they will be the same in the parent process and in the child ones. The task of the parent process with PID 15771 is to monitor the child processes, so it just waits for a response from the children.
Therefore, he does not need a pipe, and he closes the file descriptors with numbers 3 and 4.
In the first bash child process with PID 9004, the system call , changes our STDOUT file descriptor number 1 to a file descriptor pointing to a pipe, in our case it is number 3. Thus, everything that the first child process with PID 9004 writes to STDOUT will automatically fall into the pipe buffer.
In the second child process with PID 9005, bash dup2s the file to STDIN descriptor number 0. Now everything that our second bash with PID 9005 will read will read from the pipe.
After that, file descriptors with numbers 3 and 4 are also closed in child processes, since they are no longer used.
I deliberately ignore file descriptor 255, it is used internally by bash itself and will also be closed in child processes.
Next, in the first child process with PID 9004, bash starts with a system call the executable file that we specified on the command line, in our case it is /usr/bin/cat.
In the second child process with PID 9005, bash runs the second executable we specified, in our case /usr/bin/sleep.
The exec system call does not close file descriptors unless they were opened with the O_CLOEXEC flag at the time the open call was executed. In our case, after running the executable files, all current file descriptors will be saved.
Checking in the console:
[user@localhost ]$ pgrep -P 15771
9004
9005
[user@localhost ]$ ls -lah /proc/15771/fd/
total 0
dr-x------ 2 user user 0 Oct 7 15:42 .
dr-xr-xr-x 9 user user 0 Oct 7 15:42 ..
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
[user@localhost ]$ ls -lah /proc/9004/fd
total 0
dr-x------ 2 user user 0 Oct 7 15:57 .
dr-xr-xr-x 9 user user 0 Oct 7 15:57 ..
lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21
l-wx------ 1 user user 64 Oct 7 15:57 1 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
lr-x------ 1 user user 64 Oct 7 15:57 3 -> /dev/zero
[user@localhost ]$ ls -lah /proc/9005/fd
total 0
dr-x------ 2 user user 0 Oct 7 15:57 .
dr-xr-xr-x 9 user user 0 Oct 7 15:57 ..
lr-x------ 1 user user 64 Oct 7 15:57 0 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
[user@localhost ]$ ps -up 9004
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user 9004 0.0 0.0 107972 620 pts/21 S+ 15:57 0:00 cat /dev/zero
[user@localhost ]$ ps -up 9005
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user 9005 0.0 0.0 107952 360 pts/21 S+ 15:57 0:00 sleep 10000
As you can see, the unique number of our pipe is the same in both processes. Thus, we have a connection between two different processes with the same parent.
For those who are not familiar with the system calls that bash uses, I highly recommend running commands through strace and see what happens inside, for example, like this:
strace -s 1024 -f bash -c "ls | grep hello"
Let's go back to our problem with running out of disk space and trying to save data without restarting the process. Let's write a small program that will write to disk about 1 megabyte per second. Moreover, if for some reason we could not write data to disk, we will simply ignore this and try to write the data again in a second. In the example I am using Python, you can use any other programming language.
[user@localhost ]$ cat openforwrite.py
import datetime
import time
mystr="a"*1024*1024+"n"
with open("123.txt", "w") as f:
while True:
try:
f.write(str(datetime.datetime.now()))
f.write(mystr)
f.flush()
time.sleep(1)
except:
pass
Run the program and look at the file descriptors
[user@localhost ]$ python openforwrite.py &
[1] 3762
[user@localhost ]$ ps axuf | grep [o]penforwrite
user 3762 0.0 0.0 128600 5744 pts/22 S+ 16:28 0:00 | _ python openforwrite.py
[user@localhost ]$ ls -la /proc/3762/fd
total 0
dr-x------ 2 user user 0 Oct 7 16:29 .
dr-xr-xr-x 9 user user 0 Oct 7 16:29 ..
lrwx------ 1 user user 64 Oct 7 16:29 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 7 16:29 3 -> /home/user/123.txt
As you can see, we have our 3 standard file descriptors and another one that we have opened. Let's check the file size:
[user@localhost ]$ ls -lah 123.txt
-rw-rw-r-- 1 user user 117M Oct 7 16:30 123.txt
the data is written, we try to change the rights to the file:
[user@localhost ]$ sudo chown root: 123.txt
[user@localhost ]$ ls -lah 123.txt
-rw-rw-r-- 1 root root 168M Oct 7 16:31 123.txt
[user@localhost ]$ ls -lah 123.txt
-rw-rw-r-- 1 root root 172M Oct 7 16:31 123.txt
We see that the data is still being written, although our user does not have the right to write to the file. Let's try to remove it:
[user@localhost ]$ sudo rm 123.txt
[user@localhost ]$ ls 123.txt
ls: cannot access 123.txt: No such file or directory
Where is the data written? And are they written at all? We check:
[user@localhost ]$ ls -la /proc/3762/fd
total 0
dr-x------ 2 user user 0 Oct 7 16:29 .
dr-xr-xr-x 9 user user 0 Oct 7 16:29 ..
lrwx------ 1 user user 64 Oct 7 16:29 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 7 16:29 3 -> /home/user/123.txt (deleted)
Yes, our file descriptor still exists, and we can work with this file descriptor like our old file, we can read it, clean it up and copy it.
Look at the file size:
[user@localhost ]$ lsof | grep 123.txt
python 31083 user 3w REG 8,5 19923457 2621522 /home/user/123.txt
The file size is 19923457. Trying to clear the file:
[user@localhost ]$ truncate -s 0 /proc/31083/fd/3
[user@localhost ]$ lsof | grep 123.txt
python 31083 user 3w REG 8,5 136318390 2621522 /home/user/123.txt
As you can see, the file size only increases and our trunk did not work. Let's turn to the documentation on the system call . If we use the O_APPEND flag when opening a file, then with each write, the operating system checks the file size and writes data to the very end of the file, and does it atomically. This allows multiple threads or processes to write to the same file. But in our code we don't use this flag. We can see a different file size in lsof after trunk only if we open the file for writing, which means that in our code, instead of
with open("123.txt", "w") as f:
we have to put
with open("123.txt", "a") as f:
Checking with "w" flag
[user@localhost ]$ strace -e trace=open python openforwrite.py 2>&1| grep 123.txt
open("123.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
and with "a" flag
[user@localhost ]$ strace -e trace=open python openforwrite.py 2>&1| grep 123.txt
open("123.txt", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
Programming an already running process
Often, when creating and testing a program, programmers use debuggers (for example, GDB) or various levels of logging in the application. Linux provides the ability to actually write and change an already running program, such as changing the values of variables, setting a breakpoint, and so on and so forth.
Returning to the original question about the lack of disk space to write a file, let's try to simulate the problem.
Let's create a file for our partition, which we will mount as a separate drive:
[user@localhost ~]$ dd if=/dev/zero of=~/tempfile_for_article.dd bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00525929 s, 2.0 GB/s
[user@localhost ~]$
Let's create a file system:
[user@localhost ~]$ mkfs.ext4 ~/tempfile_for_article.dd
mke2fs 1.42.9 (28-Dec-2013)
/home/user/tempfile_for_article.dd is not a block special device.
Proceed anyway? (y,n) y
...
Writing superblocks and filesystem accounting information: done
[user@localhost ~]$
Let's mount the file system:
[user@localhost ~]$ sudo mount ~/tempfile_for_article.dd /mnt/
[sudo] password for user:
[user@localhost ~]$ df -h | grep mnt
/dev/loop0 8.7M 172K 7.9M 3% /mnt
Create a directory with our owner:
[user@localhost ~]$ sudo mkdir /mnt/logs
[user@localhost ~]$ sudo chown user: /mnt/logs
Let's open the file for writing only in our program:
with open("/mnt/logs/123.txt", "w") as f:
Run
[user@localhost ]$ python openforwrite.py
Waiting for a few seconds
[user@localhost ~]$ df -h | grep mnt
/dev/loop0 8.7M 8.0M 0 100% /mnt
So, we got the problem described at the beginning of this article. Free space 0, occupied 100%.
We remember that according to the conditions of the problem, we are trying to record very important data that cannot be lost. And in doing so, we need to fix the service without restarting the process.
Let's say we still have disk space, but in a different partition, for example, in / home.
Let's try to "reprogram on the fly" our code.
We look at the PID of our process, which ate all the disk space:
[user@localhost ~]$ ps axuf | grep [o]penfor
user 10078 27.2 0.0 128600 5744 pts/22 R+ 11:06 0:02 | _ python openforwrite.py
Connecting to a process with gdb
[user@localhost ~]$ gdb -p 10078
...
(gdb)
We look at open file descriptors:
(gdb) shell ls -lah /proc/10078/fd/
total 0
dr-x------ 2 user user 0 Oct 8 11:06 .
dr-xr-xr-x 9 user user 0 Oct 8 11:06 ..
lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:09 3 -> /mnt/logs/123.txt
We look at information about the file descriptor with number 3, which interests us
(gdb) shell cat /proc/10078/fdinfo/3
pos: 8189952
flags: 0100001
mnt_id: 482
Keeping in mind what system call Python does (see above where we ran strace and found an open call), while processing our code to open a file, we do the same ourselves on behalf of our process, but we need the O_WRONLY|O_CREAT|O_TRUNC bits replace with a numeric value. To do this, open the kernel sources, for example and see which flags are responsible for what
#define O_WRONLY 00000001
#define O_CREAT 00000100
#define O_TRUNC 00001000
We combine all the values \u00001101b\uXNUMXbinto one, we get XNUMX
Running our call from gdb
(gdb) call open("/home/user/123.txt", 00001101,0666)
$1 = 4
So we got a new file descriptor with number 4 and a new open file on another partition, check:
(gdb) shell ls -lah /proc/10078/fd/
total 0
dr-x------ 2 user user 0 Oct 8 11:06 .
dr-xr-xr-x 9 user user 0 Oct 8 11:06 ..
lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:09 3 -> /mnt/logs/123.txt
l-wx------ 1 user user 64 Oct 8 11:15 4 -> /home/user/123.txt
We remember the example with pipe - how bash changes file descriptors, and have already learned the dup2 system call.
Trying to replace one file descriptor with another
(gdb) call dup2(4,3)
$2 = 3
We check:
(gdb) shell ls -lah /proc/10078/fd/
total 0
dr-x------ 2 user user 0 Oct 8 11:06 .
dr-xr-xr-x 9 user user 0 Oct 8 11:06 ..
lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:09 3 -> /home/user/123.txt
l-wx------ 1 user user 64 Oct 8 11:15 4 -> /home/user/123.txt
Close file descriptor 4, since we don't need it:
(gdb) call close (4)
$1 = 0
And exit gdb
(gdb) quit
A debugging session is active.
Inferior 1 [process 10078] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/bin/python2.7, process 10078
Checking the new file:
[user@localhost ~]$ ls -lah /home/user/123.txt
-rw-rw-r-- 1 user user 5.1M Oct 8 11:18 /home/user/123.txt
[user@localhost ~]$ ls -lah /home/user/123.txt
-rw-rw-r-- 1 user user 7.1M Oct 8 11:18 /home/user/123.txt
As you can see, the data is written to a new file, we check the old one:
[user@localhost ~]$ ls -lah /mnt/logs/123.txt
-rw-rw-r-- 1 user user 7.9M Oct 8 11:08 /mnt/logs/123.txt
The data is not lost, the application works, the logs are written to a new location.
Let's make things a little more difficult
Imagine that the data is important to us, but we do not have disk space in any of the partitions and we cannot connect the disk.
What we can do is redirect our data somewhere, for example, to a pipe, and redirect the data from the pipe to the network through some program, such as netcat.
We can create a named pipe with the mkfifo command. It will create a pseudo file on the file system, even if there is no free space on it.
Restart the application and check:
[user@localhost ]$ python openforwrite.py
[user@localhost ~]$ ps axuf | grep [o]pen
user 5946 72.9 0.0 128600 5744 pts/22 R+ 11:27 0:20 | _ python openforwrite.py
[user@localhost ~]$ ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
[user@localhost ~]$ df -h | grep mnt
/dev/loop0 8.7M 8.0M 0 100% /mnt
There is no disk space, but we successfully create a named pipe there:
[user@localhost ~]$ mkfifo /mnt/logs/megapipe
[user@localhost ~]$ ls -lah /mnt/logs/megapipe
prw-rw-r-- 1 user user 0 Oct 8 11:28 /mnt/logs/megapipe
Now we need to somehow wrap all the data that gets into this pipe to another server through the network, the same netcat is suitable for this.
On the remote-server.example.com server, run
[user@localhost ~]$ nc -l 7777 > 123.txt
On our problem server, run in a separate terminal
[user@localhost ~]$ nc remote-server.example.com 7777 < /mnt/logs/megapipe
Now all the data that gets into the pipe will automatically go to stdin in netcat, which will send it to the network on port 7777.
All we have to do is start writing our data to this named pipe.
We already have a running application:
[user@localhost ~]$ ps axuf | grep [o]pen
user 5946 99.8 0.0 128600 5744 pts/22 R+ 11:27 169:27 | _ python openforwrite.py
[user@localhost ~]$ ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
Of all the flags, we only need O_WRONLY since the file already exists and we do not need to clear it
[user@localhost ~]$ gdb -p 5946
...
(gdb) call open("/mnt/logs/megapipe", 00000001,0666)
$1 = 4
(gdb) shell ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
l-wx------ 1 user user 64 Oct 8 14:20 4 -> /mnt/logs/megapipe
(gdb) call dup2(4,3)
$2 = 3
(gdb) shell ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/megapipe
l-wx------ 1 user user 64 Oct 8 14:20 4 -> /mnt/logs/megapipe
(gdb) call close(4)
$3 = 0
(gdb) shell ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/megapipe
(gdb) quit
A debugging session is active.
Inferior 1 [process 5946] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/bin/python2.7, process 5946
Checking the remote server remote-server.example.com
[user@localhost ~]$ ls -lah 123.txt
-rw-rw-r-- 1 user user 38M Oct 8 14:21 123.txt
Data is coming, we check the problem server
[user@localhost ~]$ ls -lah /mnt/logs/
total 7.9M
drwxr-xr-x 2 user user 1.0K Oct 8 11:28 .
drwxr-xr-x 4 root root 1.0K Oct 8 10:55 ..
-rw-rw-r-- 1 user user 7.9M Oct 8 14:17 123.txt
prw-rw-r-- 1 user user 0 Oct 8 14:22 megapipe
The data is saved, the problem is solved.
I take this opportunity to say hello to my colleagues from Degiro.
Listen to Radio-T podcasts.
All good.
As a homework, I propose to think about what will be in the file descriptors of the cat and sleep process if you run the following command:
[user@localhost ~]$ cat /dev/zero 2>/dev/null| sleep 10000
Source: habr.com