Downloading a 16GB torrent via a tablet with 4GB of free space

Problem:

There is a PC without the Internet, but it is possible to transfer the file via USB. There is a tablet with the Internet from which this file can be transferred. You can download the required torrent on your tablet, but there is not enough free space. The file in the torrent is one and large.

Path to solution:

I started the torrent to download. When the free space was almost gone, I paused the download. I connected the tablet to the PC and moved the file from the tablet to the PC. I paused and to my surprise the file was created again and the torrent continued downloading as if nothing had happened.

Due to the fact that the torrent client sets the sparse flag to the file in which it writes the received data, the system does not try to reserve 16GB at once and an error will not occur when trying to write to a file beyond 4GB.

After repeating the procedure four times, I received four files on my PC containing different parts of the same torrent. Now all that remains is to put them together. The procedure is essentially simple. You need to replace the zero bytes with another value if it exists at a given position in one of the four files.

It seemed to me that such a simple program should be on the Internet. Has no one ever encountered such a problem? But I realized that I don’t even know what keywords to search for it. Therefore, I quickly created a Lua script for this task and now I have optimized it. This is what I want to share.

Downloading the torrent in parts

1. start downloading the torrent on the first device
2. wait until the ROM is filled
3. pause the download
4. transfer the file to the second device and add a number to the file name
5. we return to the first point until the file is downloaded completely

Merging parts into one file

After the last part has been received, it is necessary to collect them into one whole file.

The task is simple:

1. Reading all parts at once
2. If in some part the position is not a zero byte, then we write it to the output, otherwise we write zero

Function merge_part accepts an array of threads streams_in of which reads a portion of size buffer_length and returns the result of merging parts from different threads.

function merge_part(streams_in, buffer_length)
    local out_part
    for _, stream in ipairs(streams_in) do
        local in_part = stream:read(buffer_length)

        if not out_part then
            out_part = in_part -- просто копируем часть из первого файла
        elseif in_part and #in_part > 0 then

            if #out_part < #in_part then
                out_part, in_part = in_part, out_part
            end

            if out_part ~= in_part  -- данные различаются
                and in_part:find("[^ ]")   -- есть данные в in_part
                and out_part:find(" ", 1, true) -- есть пустые места в out_part
            then 
                local find_index = 1
--[[

Function string.gsub is suitable for the task because it will find pieces filled with zeros and deliver what is given to it.

--]]
                out_part = out_part:gsub(" +", function(zero_string)

                    if #in_part < find_index then
                        return -- не на что менять
                    end
--[[

string.gsub does not convey the position at which the match was found. Therefore, we do a parallel search for the position zero_string using functions string.find. It is enough to find the first zero byte.

--]]
                    local start_index = out_part:find(" ", find_index, true)
                    find_index = start_index + #zero_string

--[[

Now if in in_part there is data for out_part copy them.

--]]
                    if #in_part >= start_index then
                        local end_index = start_index + #zero_string - 1
--[[

Cut from in_part part corresponding to the sequence of zeros.

--]]
                        local part = in_part:sub(start_index, end_index)

                        if (part:byte(1) ~= 0) or part:find("[^ ]") then
--[[

В part there is data.

--]]
                            if #part == #zero_string then
                                return part
                            else
--[[

part turned out to be less than a sequence of zeros. Let's supplement it with them.

--]]
                                return part..zero_string:sub(1, end_index - #in_part)
                            end
                        end
                    end
                end)
            end
        end
    end
    return out_part
end

Conclusion

Thus, we were able to download and assemble this file on a PC. After the merger, I pulled out the torrent file from the tablet. I installed a torrent client on my PC and checked the file with it.

The last downloaded part on the tablet can be left on the distribution, but you need to enable re-checking of the parts before this and uncheck the file so that it does not download again.

Used:

1. Flud torrent client on a tablet.
2. Torrent client qBittorent on PC.
3. Lua script

Source: habr.com