Parathënie
Ekziston njĂ« utilitar kaq i thjeshtĂ« dhe shumĂ« i dobishĂ«m nĂ« botĂ« â , dhe ndodhi qĂ« ai e ka gjetur rrugĂ«n e tij nĂ« procesin tonĂ« tĂ« prodhimit (sidoqoftĂ«, nuk arrita tĂ« instaloj versionin e saj, por ishin absolutisht jo versionet mĂ« tĂ« fundit). E pĂ«rdorim atĂ« pĂ«r qĂ«llimin e saj tĂ« drejtpĂ«rdrejtĂ« â ndĂ«rtimin e patches binarĂ«. NĂ«se e shikoni se çfarĂ« Ă«shtĂ« atje nĂ« depo, bĂ«het pak trishtues: nĂ« thelb, ajo Ă«shtĂ« braktisur qĂ« shumĂ« kohĂ« dhe shumĂ« gjĂ«ra aty janĂ« shumĂ« tĂ« vjetruara (pĂ«r njĂ« herĂ«, kolegu im i mĂ«parshĂ«m bĂ«ri disa ndryshime, por kjo ishte shumĂ« kohĂ« mĂ« parĂ«). Pra, vendosa ta ringjall kĂ«tĂ« projekt: bĂ«ra fork, hodha jashtĂ« atĂ« qĂ« nuk planifikoj tĂ« pĂ«rdor, dhe e kalova projektin nĂ« , inkorporova mikrofunksionet "e nxehta", hoqa nga staku masat e mĂ«dha (dhe masat e ndryshueshme, nga tĂ« cilat mĂ« vinte sinqerisht keq), dhe e kalova pĂ«rsĂ«ri profiler-in â dhe zbulova se rreth 40% e kohĂ«s shpenzohej pĂ«r âŠ
ĂfarĂ« ndodhi me fwrite?
NĂ« kĂ«tĂ« kod fwrite (nĂ« rastin tim tĂ« veçantĂ«: ndĂ«rtimi i patch-it midis dy skedarĂ«ve tĂ« afĂ«rt me 300 MB, tĂ« dhĂ«nat hyrĂ«se krejtĂ«sisht nĂ« memorie) thirret miliona herĂ« me njĂ« buffer tĂ« vogĂ«l. ĂshtĂ« e qartĂ« se ky gjĂ« do tĂ« ngadalĂ«sojĂ« punĂ«n, dhe prandaj do t'ia dalim qĂ« tĂ« ndikojmĂ« nĂ« kĂ«tĂ« situatĂ« pa dĂ«shiruar tĂ« implementojmĂ« lloje tĂ« ndryshme burimesh tĂ« dhĂ«nash, ose hyrje-dalje asinkron. E para qĂ« mĂ« erdhi nĂ« mendje â tĂ« rrisja madhĂ«sinĂ« e buffer-it
setvbuf(file, nullptr, _IOFBF, 64 * 1024)por nuk kam marrĂ« ndonjĂ« pĂ«rmirĂ«sim tĂ« rĂ«ndĂ«sishĂ«m (tani fwrite merrte rreth 37% tĂ« kohĂ«s) â kĂ«shtu qĂ« problemi nuk Ă«shtĂ« nĂ« shkrimin e shpeshtĂ« tĂ« tĂ« dhĂ«nave nĂ« disk. Duke shqyrtuar "nĂ«n kapak" fwrite mund tĂ« shohim se brenda ndodhin njĂ« bllokim/zhbllokim i strukturĂ«s FILE nĂ« mĂ«nyrĂ« tĂ« tillĂ« (pseudokod, gjithĂ« analiza u krye nĂ«n Visual Studio 2017):
size_t fwrite (const void *buffer, size_t size, size_t count, FILE *stream)
{
size_t retval = 0;
_lock_str(stream); /* blloko rrjedhën */
__try
{
retval = _fwrite_nolock(buffer, size, count, stream);
}
__finally
{
_unlock_str(stream); /* zhblloko rrjedhën */
}
return retval;
}
NĂ«se i besoj profiler-it, nĂ« _fwrite_nolock ndodhin vetĂ«m 6% e kohĂ«s, pĂ«r pjesĂ«n tjetĂ«r â overhead. NĂ« rastin tim tĂ« veçantĂ«, siguria e rrjedhĂ«s Ă«shtĂ« njĂ« e tepricĂ« e dukshme, prandaj do ta sakrifikoj atĂ«, duke e zĂ«vendĂ«suar thirrjen fwrite me â as even with arguments, there's no need to complicate things. In conclusion: this simple manipulation drastically reduced the costs of recording the result, which, in the original version, accounted for almost half of the time spent. By the way, there is a similar function in the POSIX world â . Generally speaking, the same goes for fread. Thus, using a couple of #define, you can achieve a fairly cross-platform solution without unnecessary locks when they are not needed (and this happens quite often).
fwrite, _fwrite_nolock, setvbuf
Let's abstract from the original project and focus on testing a specific case: writing a large file (512 MB) in extremely small portions â 1 byte at a time. Test system: AMD Ryzen 7 1700, 16 GB RAM, HDD 7200 rpm with 64 MB cache, Windows 10 1809, 32-bit binary built, optimizations enabled, library statically linked.
Sample for conducting the experiment:
#include <chrono>
#include <cstdio>
#include <inttypes.h>
#include <memory>
#ifdef _MSC_VER
#define fwrite_unlocked _fwrite_nolock
#endif
using namespace std::chrono;
int main()
{
std::unique_ptr<FILE, int(*)(FILE*)> file(fopen("test.bin", "wb"), fclose);
if (!file)
return 1;
constexpr size_t TEST_BUFFER_SIZE = 256 * 1024;
if (setvbuf(file.get(), nullptr, _IOFBF, TEST_BUFFER_SIZE) != 0)
return 2;
auto start = steady_clock::now();
const uint8_t b = 77;
constexpr size_t TEST_FILE_SIZE = 512 * 1024 * 1024;
for (size_t i = 0; i < TEST_FILE_SIZE; ++i)
fwrite_unlocked(&b, 1, sizeof(b), file.get());
auto end = steady_clock::now();
auto interval = duration_cast<microseconds>(end - start);
printf("Time: %lldn", interval.count());
return 0;
}
The variables will be TEST_BUFFER_SIZE, and for a couple of cases, we will replace fwrite_unlocked with fwrite. Let's start with the fwrite case without explicitly setting the buffer size (let's comment out setvbuf and related code): time 27048906 ”s, write speed â 18.93 MB/s. Now let's set the buffer size to 64 KB: time â 25037111 ”s, speed â 20.44 MB/s. Now we'll test _fwrite_nolock without calling setvbuf: 7262221 ”s, speed â 70.5 MB/s!
Next, let's experiment with buffer size (setvbuf):
The data was obtained by averaging 5 experiments; I was too lazy to account for inaccuracies. As for me, 93 MB/s when writing 1 byte on a regular HDD is quite a good result, you just need to choose the optimal buffer size (for me, 256 KB is just right) and replace fwrite with _fwrite_nolock/fwrite_unlocked (if thread safety is not needed, of course).
Similarly with fread under such conditions. Since I don't have a 'metal' machine with Linux at hand (single-board computers don't count), I decided to conduct a limited experiment on a virtual machine (Hyper-V, OpenSUSE 15, GCC 8.3.1) â the pattern, in principle, is the same: 'bare' fwrite gives 20 MB/s, fwrite + 256 KB buffer produced 23 MB/s, fwrite_unlocked with the same buffer â 35 MB/s (64-bit binary built with g++ -o2 -s -static-libgcc -static-libstdc++ fwrite_test.cpp -o fwrite_test).
Pasthënie
QĂ«llimi i kĂ«tij artikulli ishte pĂ«rshkrimi i njĂ« metode tĂ« thjeshtĂ« dhe efektive nĂ« shumĂ« raste (me funksionet _fwrite_nolock/fwrite_unlocked nuk kam pasur ndonjĂ«herĂ« tĂ« bĂ«j, nuk janĂ« shumĂ« tĂ« njohura â por kot). Nuk pretendoj pĂ«r novacion tĂ« materialit, por shpresoj qĂ« artikulli do tĂ« jetĂ« i dobishĂ«m pĂ«r komunitetin.
Burimi: habr.com
