Once, in an interview, I was asked what I would do if I discovered a service not working because the disk ran out of space.
Of course, I replied that I would check what was occupying that space and if possible, I would free up some space.
Then the interviewer asked, what if there is no free space on the partition, but you also don't see any files occupying all the space?
I said that you can always look at the open file descriptors, for example, with the lsof command, and understand which application has taken up all the available space, then we can act according to the circumstances, depending on whether the data is needed.
The interviewer interrupted me at the last word, adding to his question: "Suppose the data is not needed, it's just a debug log, but the application is not working because it cannot write the debug?"
"Okay," I replied, "we can turn off debug in the application's config and restart it."
The interviewer countered: "No, we cannot restart the application, we still have important data stored in memory, and important clients are connected to the service, whom we cannot force to reconnect."
"Well then," I said, "if we cannot restart the application and the data is not important to us, we can simply clean up this open file through the file descriptor, even if we can't see it with the ls command in the filesystem."
The interviewer was satisfied, but I was not.
I then thought, why is the person checking my knowledge not digging deeper? What if the data is actually important? What if we cannot restart the process, and at the same time, this process is writing to the filesystem on a partition that has no free space? What if we cannot afford to lose not only the already written data but also the data that this process is writing or trying to write?
Tuzik
At the beginning of my career, I tried to create a small application that needed to store information about users. I wondered how I could link a user to their data. For example, I have Ivanov Ivan Ivanovich, and he has some data, but how do I connect them? I can specify directly that a dog named "Tuzik" belongs to this Ivan. But what if he changes his name and becomes, for example, Olya? Then it turns out that our Olya Ivanovna Ivanova no longer has a dog, while our Tuzik still belongs to a non-existent Ivan. This problem was solved by a database, which assigned a unique identifier (ID) to each user, and my Tuzik was linked to this ID, which was essentially just a sequential number. Thus, the owner of Tuzik had an ID of 2, and at some point, that ID was Ivan, and later it became Olya. The problem of humanity and animal husbandry was practically resolved.
File descriptor
The problem of a file and the program working with that file is somewhat similar to that of our dog and human. Suppose I opened a file named ivan.txt and started writing the word tuzik in it, but I only managed to write the first letter "t" in the file before it was renamed by someone, for example, to olya.txt. But the file remains the same, and I still want to write my Tuzik in it. Every time I open the file through a system call in any programming language, I get a unique ID that points me to the file; this ID is the file descriptor. It doesn't matter what happens to this file afterward, it could be deleted, renamed, its owner could be changed, or read and write permissions could be taken away, I will still have access to it because at the moment of opening the file, I had permissions to read and/or write it, and I managed to start working on it, so I should continue doing so.
In Linux, the libc library opens 3 file descriptors for each running application (process), with numbers 0, 1, 2. You can find more information at the links and
- File descriptor 0 is called STDIN and is associated with input data for the application.
- File descriptor 1 is called STDOUT and is used by applications for outputting data, such as the print commands.
- File descriptor 2 is called STDERR and is used by applications for outputting error messages.
If you open any file for reading or writing in your program, you will most likely receive the first available ID, which will be number 3.
The list of file descriptors can be viewed for any process if you know its PID.
For example, let's open a console with bash and check the PID of our process.
[user@localhost ]$ echo $$
15771
In the second console, let's run
[user@localhost ]$ ls -lah /proc/15771/fd/
total 0
dr-x------ 2 user user 0 Oct 7 15:42 .
dr-xr-xr-x 9 user user 0 Oct 7 15:42 ..
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
You can safely ignore file descriptor number 255 in the context of this article; it has already been opened for its own purposes by bash, not the linked library.
Currently, all 3 file descriptors are linked to the pseudo-terminal device. , but we can still manipulate them, for example, let's run in the second console
[user@localhost ]$ echo "hello world" > /proc/15771/fd/0
And in the first console, we will see
[user@localhost ]$ hello world
Redirect and Pipe
You can easily override these 3 file descriptors in any process, including in bash, for example, through a pipe connecting two processes, let’s see
[user@localhost ]$ cat /dev/zero | sleep 10000
You can run this command with strace -f and see what happens inside, but I'll briefly explain.
Our parent process bash with PID 15771 parses our command and understands how many commands we want to run, in our case, there are two: cat and sleep. Bash knows that it needs to create two child processes and connect them with one pipe. So, bash will need 2 child processes and one pipe.
Before creating the child processes, bash makes a system call and obtains new file descriptors for the temporary buffer of the pipe, but this buffer does not yet link our two child processes.
For the parent process, it looks like there is already a pipe while the child processes do not yet exist:
PID command
15771 bash
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
Then, using a system call, bash creates two child processes, and our three processes will look like this:
PID command
15771 bash
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
PID command
9004 bash
lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 255 -> /dev/pts/21
PID command
9005 bash
lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 3 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 4 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 255 -> /dev/pts/21
Don't forget that clone duplicates the process along with all file descriptors, so they will be the same in both the parent process and the child ones. The parent process with PID 15771 needs to monitor the child processes, so it simply waits for a response from them.
Consequently, it doesn't need the pipe, and it closes the file descriptors numbered 3 and 4.
In the first child process bash with PID 9004, through the system call , changes our STDOUT file descriptor from number 1 to the file descriptor pointing to the pipe, which in our case is number 3. Thus, everything that the first child process with PID 9004 writes to STDOUT will automatically go to the pipe buffer.
In the second child process with PID 9005, bash changes the STDIN file descriptor from number 0 using dup2. Now everything that our second bash with PID 9005 reads will come from the pipe.
After that, the file descriptors numbered 3 and 4 are also closed in the child processes since they are no longer used.
I intentionally ignore file descriptor 255; it is used for bash's internal purposes and will also be closed in the child processes.
Next, in the first child process with PID 9004, bash initiates the system call for the executable file that we specified in the command line, in our case it is /usr/bin/cat.
In the second child process with PID 9005, bash runs the second executable file that we specified, which in our case is /usr/bin/sleep.
The exec system call does not close file descriptors if they were not opened with the O_CLOEXEC flag during the open call. In our case, after executing the files, all current file descriptors will be preserved.
Let's check in the console:
[user@localhost ]$ pgrep -P 15771
9004
9005
[user@localhost ]$ ls -lah /proc/15771/fd/
total 0
dr-x------ 2 user user 0 Oct 7 15:42 .
dr-xr-xr-x 9 user user 0 Oct 7 15:42 ..
lrwx------ 1 user user 64 Oct 7 15:42 0 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 2 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:42 255 -> /dev/pts/21
[user@localhost ]$ ls -lah /proc/9004/fd
total 0
dr-x------ 2 user user 0 Oct 7 15:57 .
dr-xr-xr-x 9 user user 0 Oct 7 15:57 ..
lrwx------ 1 user user 64 Oct 7 15:57 0 -> /dev/pts/21
l-wx------ 1 user user 64 Oct 7 15:57 1 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
lr-x------ 1 user user 64 Oct 7 15:57 3 -> /dev/zero
[user@localhost ]$ ls -lah /proc/9005/fd
total 0
dr-x------ 2 user user 0 Oct 7 15:57 .
dr-xr-xr-x 9 user user 0 Oct 7 15:57 ..
lr-x------ 1 user user 64 Oct 7 15:57 0 -> pipe:[253543032]
lrwx------ 1 user user 64 Oct 7 15:57 1 -> /dev/pts/21
lrwx------ 1 user user 64 Oct 7 15:57 2 -> /dev/pts/21
[user@localhost ]$ ps -up 9004
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user 9004 0.0 0.0 107972 620 pts/21 S+ 15:57 0:00 cat /dev/zero
[user@localhost ]$ ps -up 9005
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
user 9005 0.0 0.0 107952 360 pts/21 S+ 15:57 0:00 sleep 10000
As you can see, the unique pipe number is the same in both processes. Thus, we have a connection between two different processes with the same parent.
For those unfamiliar with the system calls that bash uses, I highly recommend running the commands through strace to see what happens internally, for example, like this:
strace -s 1024 -f bash -c "ls | grep hello"
Let's return to our issue with insufficient disk space and the attempt to save data without restarting the process. We will write a small program that writes about 1 megabyte to disk per second. If, for any reason, we are unable to write data to disk, we will simply ignore this and attempt to write the data again after a second. In the example, I am using Python, but you can use any other programming language.
[user@localhost ]$ cat openforwrite.py
import datetime
import time
mystr="a"*1024*1024+"n"
with open("123.txt", "w") as f:
while True:
try:
f.write(str(datetime.datetime.now()))
f.write(mystr)
f.flush()
time.sleep(1)
except:
pass
Let's run the program and check the file descriptors.
[user@localhost ]$ python openforwrite.py &
[1] 3762
[user@localhost ]$ ps axuf | grep [o]penforwrite
user 3762 0.0 0.0 128600 5744 pts/22 S+ 16:28 0:00 | _ python openforwrite.py
[user@localhost ]$ ls -la /proc/3762/fd
total 0
dr-x------ 2 user user 0 Oct 7 16:29 .
dr-xr-xr-x 9 user user 0 Oct 7 16:29 ..
lrwx------ 1 user user 64 Oct 7 16:29 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 7 16:29 3 -> /home/user/123.txt
As we can see, we have our 3 standard file descriptors and one additional one that we've opened. Let's check the file size:
[user@localhost ]$ ls -lah 123.txt
-rw-rw-r-- 1 user user 117M Oct 7 16:30 123.txt
Data is being written, let's try to change the permissions on the file:
[user@localhost ]$ sudo chown root: 123.txt
[user@localhost ]$ ls -lah 123.txt
-rw-rw-r-- 1 root root 168M Oct 7 16:31 123.txt
[user@localhost ]$ ls -lah 123.txt
-rw-rw-r-- 1 root root 172M Oct 7 16:31 123.txt
We see that data is still being written, even though our user does not have permission to write to the file. Let's try to delete it:
[user@localhost ]$ sudo rm 123.txt
[user@localhost ]$ ls 123.txt
ls: cannot access 123.txt: No such file or directory
Where is the data being written? And is it being written at all? Let's check:
[user@localhost ]$ ls -la /proc/3762/fd
total 0
dr-x------ 2 user user 0 Oct 7 16:29 .
dr-xr-xr-x 9 user user 0 Oct 7 16:29 ..
lrwx------ 1 user user 64 Oct 7 16:29 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 7 16:29 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 7 16:29 3 -> /home/user/123.txt (deleted)
Yes, our file descriptor still exists, and we can work with this file descriptor as if it were our old file; we can read from it, clear it, and copy it.
Let's look at the file size:
[user@localhost ]$ lsof | grep 123.txt
python 31083 user 3w REG 8,5 19923457 2621522 /home/user/123.txt
The file size is 19923457. Let's try to clear the file:
[user@localhost ]$ truncate -s 0 /proc/31083/fd/3
[user@localhost ]$ lsof | grep 123.txt
python 31083 user 3w REG 8,5 136318390 2621522 /home/user/123.txt
As we can see, the file size only increases and our truncate did not work. Let’s refer to the documentation for system calls. If we use the O_APPEND flag when opening a file, then when we write data, the operating system checks the file size and writes data at the very end of the file, doing so atomically. This allows multiple threads or processes to write to the same file. However, in our code, we are not using this flag. We can see a different file size in lsof after truncate only if we open the file for appending. Therefore, in our code, instead of
with open("123.txt", "w") as f:
we should write
with open("123.txt", "a") as f:
Checking with the "w" flag
[user@localhost ]$ strace -e trace=open python openforwrite.py 2>&1 | grep 123.txt
open("123.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
and with the 'a' flag
[user@localhost ]$ strace -e trace=open python openforwrite.py 2>&1 | grep 123.txt
open("123.txt", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3
Programming an already running process
Often, programmers use debuggers (such as GDB) or different logging levels in applications during the creation and testing of a program. Linux actually provides the ability to write and modify an already running program, for instance, changing variable values, setting breakpoints, and so on.
Returning to the original question about the lack of disk space to write a file, let's try to simulate the problem.
We will create a file for our section, which we will mount as a separate disk:
[user@localhost ~]$ dd if=/dev/zero of=~/tempfile_for_article.dd bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.00525929 s, 2.0 GB/s
[user@localhost ~]$
Let's create a filesystem:
[user@localhost ~]$ mkfs.ext4 ~/tempfile_for_article.dd
mke2fs 1.42.9 (28-Dec-2013)
/home/user/tempfile_for_article.dd is not a block special device.
Proceed anyway? (y,n) y
...
Writing superblocks and filesystem accounting information: done
[user@localhost ~]$
Let's mount the filesystem:
[user@localhost ~]$ sudo mount ~/tempfile_for_article.dd /mnt/
[sudo] password for user:
[user@localhost ~]$ df -h | grep mnt
/dev/loop0 8.7M 172K 7.9M 3% /mnt
Creating a directory with our owner:
[user@localhost ~]$ sudo mkdir /mnt/logs
[user@localhost ~]$ sudo chown user: /mnt/logs
We will open the file for writing only in our program:
with open("/mnt/logs/123.txt", "w") as f:
Launch
[user@localhost ]$ python openforwrite.py
Waiting a few seconds
[user@localhost ~]$ df -h | grep mnt
/dev/loop0 8.7M 8.0M 0 100% /mnt
Thus, we encountered the problem described at the beginning of this article. Free space 0, used 100%.
We remember that according to the conditions of the task, we are trying to write very important data that cannot be lost. And at the same time, we need to fix the service without restarting the process.
Suppose we still have disk space, but in another partition, for example, in /home.
Let's try to 'hot reprogram' our code.
Looking at the PID of our process that has consumed all the disk space:
[user@localhost ~]$ ps axuf | grep [o]penfor
user 10078 27.2 0.0 128600 5744 pts/22 R+ 11:06 0:02 | _ python openforwrite.py
Connecting to the process via gdb
[user@localhost ~]$ gdb -p 10078
...
(gdb)
Looking at the open file descriptors:
(gdb) shell ls -lah /proc/10078/fd/
total 0
dr-x------ 2 user user 0 Oct 8 11:06 .
dr-xr-xr-x 9 user user 0 Oct 8 11:06 ..
lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:09 3 -> /mnt/logs/123.txt
Looking at the information of file descriptor number 3, which is of interest to us
(gdb) shell cat /proc/10078/fdinfo/3
pos: 8189952
flags: 0100001
mnt_id: 482
Remembering the system call that Python makes (see above where we ran strace and found the open call), while handling our code to open a file, we do the same ourselves on behalf of our process, but we need to replace the bits O_WRONLY|O_CREAT|O_TRUNC with a numeric value. For this, we open the kernel source code, for example, and check what each flag stands for.
#define O_WRONLY 00000001
#define O_CREAT 00000100
#define O_TRUNC 00001000
We combine all the values into one, getting 00001101.
Now we run our call from gdb.
(gdb) call open("/home/user/123.txt", 00001101,0666)
$1 = 4
So we got a new file descriptor with number 4 and a new opened file on a different partition, let's check:
(gdb) shell ls -lah /proc/10078/fd/
total 0
dr-x------ 2 user user 0 Oct 8 11:06 .
dr-xr-xr-x 9 user user 0 Oct 8 11:06 ..
lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:09 3 -> /mnt/logs/123.txt
l-wx------ 1 user user 64 Oct 8 11:15 4 -> /home/user/123.txt
We remember the example with pipe — how bash changes file descriptors, and we've already learned the system call dup2.
We try to replace one file descriptor with another.
(gdb) call dup2(4,3)
$2 = 3
Checking:
(gdb) shell ls -lah /proc/10078/fd/
total 0
dr-x------ 2 user user 0 Oct 8 11:06 .
dr-xr-xr-x 9 user user 0 Oct 8 11:06 ..
lrwx------ 1 user user 64 Oct 8 11:09 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:09 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:06 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:09 3 -> /home/user/123.txt
l-wx------ 1 user user 64 Oct 8 11:15 4 -> /home/user/123.txt
We close file descriptor 4, as we no longer need it:
(gdb) call close (4)
$1 = 0
And we exit gdb.
(gdb) quit
A debugging session is active.
Inferior 1 [process 10078] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/bin/python2.7, process 10078
We check the new file:
[user@localhost ~]$ ls -lah /home/user/123.txt
-rw-rw-r-- 1 user user 5.1M Oct 8 11:18 /home/user/123.txt
[user@localhost ~]$ ls -lah /home/user/123.txt
-rw-rw-r-- 1 user user 7.1M Oct 8 11:18 /home/user/123.txt
As we can see, data is being written to the new file, let's check the old one:
[user@localhost ~]$ ls -lah /mnt/logs/123.txt
-rw-rw-r-- 1 user user 7.9M Oct 8 11:08 /mnt/logs/123.txt
Data has not been lost, the application is running, logs are being written to a new location.
Let's complicate the task a bit.
Suppose our data is important, but we have no space on the disk in any of the partitions and we cannot attach a disk.
What we can do is redirect our data somewhere, for example into a pipe, and then redirect the data from the pipe into the network through some program, like netcat.
We can create a named pipe using the mkfifo command. It will create a pseudo-file on the file system, even if there is no free space on it.
Restarting the application and checking:
[user@localhost ]$ python openforwrite.py
[user@localhost ~]$ ps axuf | grep [o]pen
user 5946 72.9 0.0 128600 5744 pts/22 R+ 11:27 0:20 | _ python openforwrite.py
[user@localhost ~]$ ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
[user@localhost ~]$ df -h | grep mnt
/dev/loop0 8.7M 8.0M 0 100% /mnt
There is no disk space, but we successfully create a named pipe there:
[user@localhost ~]$ mkfifo /mnt/logs/megapipe
[user@localhost ~]$ ls -lah /mnt/logs/megapipe
prw-rw-r-- 1 user user 0 Oct 8 11:28 /mnt/logs/megapipe
Now we need to somehow wrap all the data that flows into this pipe to another server over the network, and netcat will do just fine.
On the server remote-server.example.com, we start
[user@localhost ~]$ nc -l 7777 > 123.txt
On our problematic server, we run in a separate terminal
[user@localhost ~]$ nc remote-server.example.com 7777 < /mnt/logs/megapipe
Now all data that goes into the pipe will automatically be redirected to stdin in netcat, which will send it across the network to port 7777.
All we need to do now is to start writing our data to this named pipe.
We already have a running application:
[user@localhost ~]$ ps axuf | grep [o]pen
user 5946 99.8 0.0 128600 5744 pts/22 R+ 11:27 169:27 | _ python openforwrite.py
[user@localhost ~]$ ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
Of all the flags, we only need O_WRONLY since the file already exists and we don't need to clear it.
[user@localhost ~]$ gdb -p 5946
...
(gdb) call open("/mnt/logs/megapipe", 00000001,0666)
$1 = 4
(gdb) shell ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/123.txt
l-wx------ 1 user user 64 Oct 8 14:20 4 -> /mnt/logs/megapipe
(gdb) call dup2(4,3)
$2 = 3
(gdb) shell ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/megapipe
l-wx------ 1 user user 64 Oct 8 14:20 4 -> /mnt/logs/megapipe
(gdb) call close(4)
$3 = 0
(gdb) shell ls -lah /proc/5946/fd
total 0
dr-x------ 2 user user 0 Oct 8 11:27 .
dr-xr-xr-x 9 user user 0 Oct 8 11:27 ..
lrwx------ 1 user user 64 Oct 8 11:28 0 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:28 1 -> /dev/pts/22
lrwx------ 1 user user 64 Oct 8 11:27 2 -> /dev/pts/22
l-wx------ 1 user user 64 Oct 8 11:28 3 -> /mnt/logs/megapipe
(gdb) quit
A debugging session is active.
Inferior 1 [process 5946] will be detached.
Quit anyway? (y or n) y
Detaching from program: /usr/bin/python2.7, process 5946
Checking the remote server remote-server.example.com
[user@localhost ~]$ ls -lah 123.txt
-rw-rw-r-- 1 user user 38M Oct 8 14:21 123.txt
Data is flowing, checking the problematic server
[user@localhost ~]$ ls -lah /mnt/logs/
total 7.9M
drwxr-xr-x 2 user user 1.0K Oct 8 11:28 .
drwxr-xr-x 4 root root 1.0K Oct 8 10:55 ..
-rw-rw-r-- 1 user user 7.9M Oct 8 14:17 123.txt
prw-rw-r-- 1 user user 0 Oct 8 14:22 megapipe
Data has been saved, the problem is solved.
Taking this opportunity to say hello to my colleagues at Degiro.
Listen to the Radio-T podcasts.
Wishing everyone well.
As homework, I suggest thinking about what will be in the file descriptors for the processes of cat and sleep if you run the following command:
[user@localhost ~]$ cat /dev/zero 2>/dev/null| sleep 10000
Source: habr.com
