Speeding up C/C++ file I/O without too much effort

foreword

There is such a simple and very useful utility in the world - BDelta, and it so happened that it took root in our production process for a very long time (although we could not install its version, but it certainly was not the last one available). We use it for its intended purpose - the construction of binary patches. If you look at what is in the repository, it becomes a little sad: in fact, it was abandoned a long time ago and much of it is very outdated (once my former colleague made several edits there, but it was a long time ago). In general, I decided to resurrect this business: I forked, threw out what I did not plan to use, overtook the project to cmake, inlined “hot” microfunctions, removed large arrays from the stack (and arrays of variable length, from which I frankly “bomb”), ran the profiler once again - and found out that about 40% of the time is spent on fwrite...

So what's up with fwrite?

In this code, fwrite (in my particular test case: building a patch between close 300MB files, input data entirely in memory) is called millions of times with a small buffer. Obviously, this thing will slow down, and therefore I would like to somehow influence this disgrace. There is no desire to introduce various kinds of data sources, asynchronous I / O, I wanted to find a simpler solution. The first thing that came to mind was to increase the buffer size

setvbuf(file, nullptr, _IOFBF, 64* 1024)

but I didn’t get a significant improvement in the result (now fwrite accounted for about 37% of the time) - it means that the point is still not in the frequent writing of data to disk. Looking “under the hood” of fwrite, you can see what happens inside the lock / unlock FILE structure like this (pseudocode, all analysis was carried out under Visual Studio 2017):

size_t fwrite (const void *buffer, size_t size, size_t count, FILE *stream)
{
   size_t retval = 0;
   _lock_str(stream);   /* lock stream */
   __try
   {
      retval = _fwrite_nolock(buffer, size, count, stream);
   }
   __finally 
   {
       _unlock_str(stream);   /* unlock stream */
   }
   return retval;
}

According to the profiler, _fwrite_nolock accounts for only 6% of the time, the rest is overhead. In my particular case, thread safety is an obvious overkill, and I will sacrifice it by replacing the fwrite call with _fwrite_nolock You don't even need to be smart with arguments. Total: this simple manipulation significantly reduced the cost of recording the result, which in the original version was almost half the time spent. By the way, in the POSIX world there is a similar function − fwrite_unlocked. Generally speaking, the same applies to fread. Thus, using the #define pair, you can get a completely cross-platform solution without unnecessary locks if they are not needed (and this happens very often).

fwrite, _fwrite_nolock, setvbuf

Let's move away from the original project and focus on testing a specific case: writing a large file (512 MB) in extremely small chunks—one byte each. Test system: AMD Ryzen 7 1700, 16 GB RAM, 7200 rpm HDD, 64 MB cache. Windows 10 1809, the binary was built as 32-bit, optimizations are enabled, the library is statically linked.

Sample for the experiment:

#include <chrono>
#include <cstdio>
#include <inttypes.h>
#include <memory>

#ifdef _MSC_VER
#define fwrite_unlocked _fwrite_nolock
#endif

using namespace std::chrono;

int main()
{
    std::unique_ptr<FILE, int(*)(FILE*)> file(fopen("test.bin", "wb"), fclose);
    if (!file)
        return 1;

    constexpr size_t TEST_BUFFER_SIZE = 256 * 1024;
    if (setvbuf(file.get(), nullptr, _IOFBF, TEST_BUFFER_SIZE) != 0)
        return 2;

    auto start = steady_clock::now();
    const uint8_t b = 77;
    constexpr size_t TEST_FILE_SIZE = 512 * 1024 * 1024;
    for (size_t i = 0; i < TEST_FILE_SIZE; ++i)
        fwrite_unlocked(&b, 1, sizeof(b), file.get());

    auto end = steady_clock::now();
    auto interval = duration_cast<microseconds>(end - start);
    printf("Time: %lldn", interval.count());

    return 0;
}

TEST_BUFFER_SIZE will act as variables, and for a couple of cases we will replace fwrite_unlocked with fwrite. Let's start with the case of fwrite without explicitly setting the buffer size (comment out setvbuf and related code): time 27048906 µs, write speed 18.93 MB/s. Now let's set the buffer size to 64 KB: time - 25037111 µs, speed - 20.44 Mb / s. Now let's test the operation of _fwrite_nolock without calling setvbuf: 7262221 µs, speed - 70.5 Mb / s!

Next, let's experiment with the buffer size (setvbuf):

The data were obtained by averaging 5 experiments, I was too lazy to count the errors. As for me, 93 Mb / s when writing 1 byte to a regular HDD is a very good result, you just need to choose the optimal buffer size (in my case, 256 Kb is just right) and replace fwrite with _fwrite_nolock / fwrite_unlocked (in if you don't need thread safety, of course).
Likewise with fread under similar conditions. Since there is no "iron" machine with linux at hand (single-boarders do not count), I decided to conduct a limited experiment on a virtual machine (Hyper-V, OpenSUSE 15, GCC 8.3.1) - the pattern is, in principle, the same: "naked" fwrite 20 MB/s, fwrite + 256 KB buffer gave 23 MB/s, fwrite_unlocked with the same buffer - 35 MB/s (64-bit binary, compiled by g++ -o2 -s -static-libgcc -static-libstdc++ fwrite_test. cpp -o fwrite_test).

Afterword

The purpose of writing this article was to describe a simple and effective technique in many cases (I somehow did not come across the _fwrite_nolock / fwrite_unlocked functions before, they are not very popular - but in vain). I do not pretend to the novelty of the material, but I hope that the article will be useful to the community.

Source: habr.com