Speeding up File I/O in C/C++ Without Too Much Effort

Preface

There is a simple and very useful utility in the world — BDelta, and it turned out that it has long been ingrained in our production process (though I couldn't install its version, it definitely wasn't the latest available). We use it for its intended purpose — creating binary patches. Looking at what’s in the repository, it’s a bit sad: it has essentially been abandoned for a long time, and much of it is significantly outdated (at one time, my former colleague made a few updates there, but that was a long time ago). In general, I decided to revive this matter: I forked it, removed what I didn’t plan to use, and migrated the project to cmake, inlined the "hot" microfunctions, removed large arrays from the stack (and variable-length arrays, which frankly annoy me), ran the profiler once again — and found that about 40% of the time is spent on fwrite…

So what's up with fwrite?

In this code, fwrite (in my specific test case: creating a patch between two similar 300 MB files, with the inputs fully in memory) is called millions of times with a small buffer. Obviously, this thing will slow down, and I would like to influence this issue somehow without introducing various data sources or asynchronous I/O for now; I wanted to find a simpler solution. The first thing that came to mind was to increase the buffer size

setvbuf(file, nullptr, _IOFBF, 64* 1024)

but I didn't see a significant improvement in results (now fwrite accounted for about 37% of the time) — so the issue is indeed not with frequent disk writes. Taking a look "under the hood" of fwrite, you can see that internally it involves locking/unlocking the FILE structure roughly like this (pseudocode, all analysis was conducted under Visual Studio 2017):

size_t fwrite (const void *buffer, size_t size, size_t count, FILE *stream)
{
   size_t retval = 0;
   _lock_str(stream);   /* lock stream */
   __try
   {
      retval = _fwrite_nolock(buffer, size, count, stream);
   }
   __finally 
   {
       _unlock_str(stream);   /* unlock stream */
   }
   return retval;
}

If the profiler is to be believed, _fwrite_nolock takes only 6% of the time, the rest is overhead. In my specific case, thread safety is clearly superfluous, and I will sacrifice it by replacing the fwrite call with _fwrite_nolock — even with arguments, there's no need for complications. In summary, this simple manipulation has significantly reduced the costs of writing the results, which in the original version accounted for almost half of the total time spent. By the way, in the world of POSIX, there is a similar function — fwrite_unlocked. Generally speaking, the same applies to fread. Thus, with a couple of #define, one can achieve a fairly cross-platform solution without unnecessary locks when they're not needed (which often happens).

fwrite, _fwrite_nolock, setvbuf

Let's abstract from the original project and focus on testing a specific case: writing a large file (512 MB) in extremely small portions — 1 byte at a time. Test system: AMD Ryzen 7 1700, 16 GB RAM, 7200 rpm HDD with 64 MB cache, Windows 10 1809, the binary was built 32-bit, optimizations included, and the library is statically linked.

Sample for conducting the experiment:

#include <chrono>
#include <cstdio>
#include <inttypes.h>
#include <memory>

#ifdef _MSC_VER
#define fwrite_unlocked _fwrite_nolock
#endif

using namespace std::chrono;

int main()
{
    std::unique_ptr<FILE, int(*)(FILE*)> file(fopen("test.bin", "wb"), fclose);
    if (!file)
        return 1;

    constexpr size_t TEST_BUFFER_SIZE = 256 * 1024;
    if (setvbuf(file.get(), nullptr, _IOFBF, TEST_BUFFER_SIZE) != 0)
        return 2;

    auto start = steady_clock::now();
    const uint8_t b = 77;
    constexpr size_t TEST_FILE_SIZE = 512 * 1024 * 1024;
    for (size_t i = 0; i < TEST_FILE_SIZE; ++i)
        fwrite_unlocked(&b, 1, sizeof(b), file.get());

    auto end = steady_clock::now();
    auto interval = duration_cast<microseconds>(end - start);
    printf("Time: %lldn", interval.count());

    return 0;
}

The variables will be TEST_BUFFER_SIZE, and for a couple of cases, we'll replace fwrite_unlocked with fwrite. We begin with the case of fwrite without explicitly setting the buffer size (we'll comment out setvbuf and the related code): time 27048906 µs, write speed — 18.93 MB/s. Now let's set the buffer size to 64 KB: time — 25037111 µs, speed — 20.44 MB/s. Now we will test _fwrite_nolock without calling setvbuf: 7262221 µs, speed — 70.5 MB/s!

Next, we will experiment with the buffer size (setvbuf):

The data is obtained by averaging 5 experiments, I was too lazy to calculate the errors. In my opinion, 93 MB/s when writing 1 byte on a regular HDD is a pretty good result; you just need to choose the optimal buffer size (in my case, 256 KB is just right) and replace fwrite with _fwrite_nolock/fwrite_unlocked (if thread safety is not required, of course).
Similarly with fread under similar conditions. Since I don't have a 'real' machine with Linux at hand (single-board computers don't count), I decided to conduct a limited experiment on a virtual machine (Hyper-V, OpenSUSE 15, GCC 8.3.1) — the pattern is generally the same: plain fwrite 20 MB/s, fwrite + 256 KB buffer gave 23 MB/s, fwrite_unlocked with the same buffer — 35 MB/s (the binary is 64-bit, built with g++ -o2 -s -static-libgcc -static-libstdc++ fwrite_test.cpp -o fwrite_test).

Afterword

The purpose of this article is to describe a simple and effective technique that is useful in many cases (I haven't encountered functions like _fwrite_nolock/fwrite_unlocked before; they are not very popular—yet they should be). I do not claim originality in the material, but I hope that this article will be beneficial to the community.

Source: habr.com