{"id":42134,"date":"2019-03-18T00:00:00","date_gmt":"2019-03-17T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas"},"modified":"2020-02-18T13:44:56","modified_gmt":"2020-02-18T10:44:56","slug":"uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","title":{"rendered":"Rritja e p\u00ebrshpejtimit t\u00eb futjes\/sakrifikimit t\u00eb skedar\u00ebve C\\\/C++, pa u lodhur shum\u00eb","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Rritja<\/p>\n

Parath\u00ebnie<\/h4>\n

\nEkziston nj\u00eb utilitar kaq i thjesht\u00eb dhe shum\u00eb i dobish\u00ebm n\u00eb bot\u00eb \u2014 BDelta<\/a><\/noindex>, dhe ndodhi q\u00eb ai e ka gjetur rrug\u00ebn e tij n\u00eb procesin ton\u00eb t\u00eb prodhimit (sidoqoft\u00eb, nuk arrita t\u00eb instaloj versionin e saj, por ishin absolutisht jo versionet m\u00eb t\u00eb fundit). E p\u00ebrdorim at\u00eb p\u00ebr q\u00ebllimin e saj t\u00eb drejtp\u00ebrdrejt\u00eb \u2014 nd\u00ebrtimin e patches binar\u00eb. N\u00ebse e shikoni se \u00e7far\u00eb \u00ebsht\u00eb atje n\u00eb depo, b\u00ebhet pak trishtues: n\u00eb thelb, ajo \u00ebsht\u00eb braktisur q\u00eb shum\u00eb koh\u00eb dhe shum\u00eb gj\u00ebra aty jan\u00eb shum\u00eb t\u00eb vjetruara (p\u00ebr nj\u00eb her\u00eb, kolegu im i m\u00ebparsh\u00ebm b\u00ebri disa ndryshime, por kjo ishte shum\u00eb koh\u00eb m\u00eb par\u00eb). Pra, vendosa ta ringjall k\u00ebt\u00eb projekt: b\u00ebra fork, hodha jasht\u00eb at\u00eb q\u00eb nuk planifikoj t\u00eb p\u00ebrdor, dhe e kalova projektin n\u00eb cmake<\/a><\/noindex>, inkorporova mikrofunksionet \"e nxehta\", hoqa nga staku masat e m\u00ebdha (dhe masat e ndryshueshme, nga t\u00eb cilat m\u00eb vinte sinqerisht keq), dhe e kalova p\u00ebrs\u00ebri profiler-in \u2014 dhe zbulova se rreth 40% e koh\u00ebs shpenzohej p\u00ebr fwrite<\/a><\/noindex>\u2026
\n<\/a><\/noindex><\/p>\n

\u00c7far\u00eb ndodhi me fwrite?<\/h4>\n

\nN\u00eb k\u00ebt\u00eb kod fwrite (n\u00eb rastin tim t\u00eb ve\u00e7ant\u00eb: nd\u00ebrtimi i patch-it midis dy skedar\u00ebve t\u00eb af\u00ebrt me 300 MB, t\u00eb dh\u00ebnat hyr\u00ebse krejt\u00ebsisht n\u00eb memorie) thirret miliona her\u00eb me nj\u00eb buffer t\u00eb vog\u00ebl. \u00cbsht\u00eb e qart\u00eb se ky gj\u00eb do t\u00eb ngadal\u00ebsoj\u00eb pun\u00ebn, dhe prandaj do t'ia dalim q\u00eb t\u00eb ndikojm\u00eb n\u00eb k\u00ebt\u00eb situat\u00eb pa d\u00ebshiruar t\u00eb implementojm\u00eb lloje t\u00eb ndryshme burimesh t\u00eb dh\u00ebnash, ose hyrje-dalje asinkron. E para q\u00eb m\u00eb erdhi n\u00eb mendje \u2014 t\u00eb rrisja madh\u00ebsin\u00eb e buffer-it<\/p>\n

setvbuf(file, nullptr, _IOFBF, 64 * 1024)<\/code><\/pre>\n
\npor nuk kam marr\u00eb ndonj\u00eb p\u00ebrmir\u00ebsim t\u00eb r\u00ebnd\u00ebsish\u00ebm (tani fwrite merrte rreth 37% t\u00eb koh\u00ebs) \u2014 k\u00ebshtu q\u00eb problemi nuk \u00ebsht\u00eb n\u00eb shkrimin e shpesht\u00eb t\u00eb t\u00eb dh\u00ebnave n\u00eb disk. Duke shqyrtuar \"n\u00ebn kapak\" fwrite mund t\u00eb shohim se brenda ndodhin nj\u00eb bllokim\/zhbllokim i struktur\u00ebs FILE n\u00eb m\u00ebnyr\u00eb t\u00eb till\u00eb (pseudokod, gjith\u00eb analiza u krye n\u00ebn Visual Studio 2017):<\/p>\n
\nsize_t fwrite (const void *buffer, size_t size, size_t count, FILE *stream)\n{\n   size_t retval = 0;\n   _lock_str(stream);   \/* blloko rrjedh\u00ebn *\/\n   __try\n   {\n      retval = _fwrite_nolock(buffer, size, count, stream);\n   }\n   __finally \n   {\n       _unlock_str(stream);   \/* zhblloko rrjedh\u00ebn *\/\n   }\n   return retval;\n}\n<\/code><\/pre>\n
\nN\u00ebse i besoj profiler-it, n\u00eb _fwrite_nolock ndodhin vet\u00ebm 6% e koh\u00ebs, p\u00ebr pjes\u00ebn tjet\u00ebr \u2014 overhead. N\u00eb rastin tim t\u00eb ve\u00e7ant\u00eb, siguria e rrjedh\u00ebs \u00ebsht\u00eb nj\u00eb e tepric\u00eb e dukshme, prandaj do ta sakrifikoj at\u00eb, duke e z\u00ebvend\u00ebsuar thirrjen fwrite me _fwrite_nolock<\/a><\/noindex> \u2014 as even with arguments, there's no need to complicate things. In conclusion: this simple manipulation drastically reduced the costs of recording the result, which, in the original version, accounted for almost half of the time spent. By the way, there is a similar function in the POSIX world \u2014 fwrite_unlocked<\/a><\/noindex>. Generally speaking, the same goes for fread. Thus, using a couple of #define, you can achieve a fairly cross-platform solution without unnecessary locks when they are not needed (and this happens quite often).<\/p>\n
fwrite, _fwrite_nolock, setvbuf<\/h4>\n
\nLet's abstract from the original project and focus on testing a specific case: writing a large file (512 MB) in extremely small portions \u2014 1 byte at a time. Test system: AMD Ryzen 7 1700, 16 GB RAM, HDD 7200 rpm with 64 MB cache, Windows 10 1809, 32-bit binary built, optimizations enabled, library statically linked.<\/p>\n
Sample for conducting the experiment:<\/p>\n
\n#include <chrono>\n#include <cstdio>\n#include <inttypes.h>\n#include <memory>\n\n#ifdef _MSC_VER\n#define fwrite_unlocked _fwrite_nolock\n#endif\n\nusing namespace std::chrono;\n\nint main()\n{\n    std::unique_ptr<FILE, int(*)(FILE*)> file(fopen(\"test.bin\", \"wb\"), fclose);\n    if (!file)\n        return 1;\n\n    constexpr size_t TEST_BUFFER_SIZE = 256 * 1024;\n    if (setvbuf(file.get(), nullptr, _IOFBF, TEST_BUFFER_SIZE) != 0)\n        return 2;\n\n    auto start = steady_clock::now();\n    const uint8_t b = 77;\n    constexpr size_t TEST_FILE_SIZE = 512 * 1024 * 1024;\n    for (size_t i = 0; i < TEST_FILE_SIZE; ++i)\n        fwrite_unlocked(&b, 1, sizeof(b), file.get());\n\n    auto end = steady_clock::now();\n    auto interval = duration_cast<microseconds>(end - start);\n    printf(\"Time: %lldn\", interval.count());\n\n    return 0;\n}\n<\/code><\/pre>\n
\nThe variables will be TEST_BUFFER_SIZE, and for a couple of cases, we will replace fwrite_unlocked with fwrite. Let's start with the fwrite case without explicitly setting the buffer size (let's comment out setvbuf and related code): time 27048906 \u00b5s, write speed \u2014 18.93 MB\/s. Now let's set the buffer size to 64 KB: time \u2014 25037111 \u00b5s, speed \u2014 20.44 MB\/s. Now we'll test _fwrite_nolock without calling setvbuf: 7262221 \u00b5s, speed \u2014 70.5 MB\/s! <\/p>\n
Next, let's experiment with buffer size (setvbuf):<\/p>\n
\"Rritja
\n
\nThe data was obtained by averaging 5 experiments; I was too lazy to account for inaccuracies. As for me, 93 MB\/s when writing 1 byte on a regular HDD is quite a good result, you just need to choose the optimal buffer size (for me, 256 KB is just right) and replace fwrite with _fwrite_nolock\/fwrite_unlocked (if thread safety is not needed, of course). 
\nSimilarly with fread under such conditions. Since I don't have a 'metal' machine with Linux at hand (single-board computers don't count), I decided to conduct a limited experiment on a virtual machine (Hyper-V, OpenSUSE 15, GCC 8.3.1) \u2014 the pattern, in principle, is the same: 'bare' fwrite gives 20 MB\/s, fwrite + 256 KB buffer produced 23 MB\/s, fwrite_unlocked with the same buffer \u2014 35 MB\/s (64-bit binary built with g++ -o2 -s -static-libgcc -static-libstdc++ fwrite_test.cpp -o fwrite_test).<\/p>\n
Pasth\u00ebnie<\/h4>\n
\nQ\u00ebllimi i k\u00ebtij artikulli ishte p\u00ebrshkrimi i nj\u00eb metode t\u00eb thjesht\u00eb dhe efektive n\u00eb shum\u00eb raste (me funksionet _fwrite_nolock\/fwrite_unlocked nuk kam pasur ndonj\u00ebher\u00eb t\u00eb b\u00ebj, nuk jan\u00eb shum\u00eb t\u00eb njohura \u2013 por kot). Nuk pretendoj p\u00ebr novacion t\u00eb materialit, por shpresoj q\u00eb artikulli do t\u00eb jet\u00eb i dobish\u00ebm p\u00ebr komunitetin.<\/p>\n
Burimi: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041f\u0440\u0435\u0434\u0438\u0441\u043b\u043e\u0432\u0438\u0435 \u0415\u0441\u0442\u044c \u043d\u0430 \u0441\u0432\u0435\u0442\u0435 \u0442\u0430\u043a\u0430\u044f \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0438 \u043e\u0447\u0435\u043d\u044c \u043f\u043e\u043b\u0435\u0437\u043d\u0430\u044f \u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u2014 BDelta, \u0438 \u0442\u0430\u043a \u0432\u044b\u0448\u043b\u043e, \u0447\u0442\u043e \u043e\u043d\u0430 \u043e\u0447\u0435\u043d\u044c \u0434\u0430\u0432\u043d\u043e \u0443\u043a\u043e\u0440\u0435\u043d\u0438\u043b\u0430\u0441\u044c \u0432 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435 (\u043f\u0440\u0430\u0432\u0434\u0430 \u0435\u0451 \u0432\u0435\u0440\u0441\u0438\u044e \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c \u043d\u0435 \u0443\u0434\u0430\u043b\u043e\u0441\u044c, \u043d\u043e \u043e\u043d\u0430 \u0442\u043e\u0447\u043d\u043e \u0431\u044b\u043b\u0430 \u043d\u0435 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0435\u0439 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0439). \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u0435\u0451 \u043f\u043e \u043f\u0440\u044f\u043c\u043e\u043c\u0443 \u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044e \u2014 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u0435 \u0431\u0438\u043d\u0430\u0440\u043d\u044b\u0445 \u043f\u0430\u0442\u0447\u0435\u0439. \u0415\u0441\u043b\u0438 \u0432\u0437\u0433\u043b\u044f\u043d\u0443\u0442\u044c, \u0447\u0442\u043e \u0442\u0430\u043c \u0432 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438, \u2014 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u0441\u043b\u0435\u0433\u043a\u0430 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-42134","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47P\u00ebrshpejtimi i hyrjes\/daljes s\u00eb skedar\u00ebve C\/C++, pa u munduar shum\u00eb | ProHoster","description":"Parath\u00ebnie Ka.","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0423\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u0435 \u0444\u0430\u0439\u043b\u043e\u0432\u043e\u0433\u043e \u0432\u0432\u043e\u0434\u0430-\u0432\u044b\u0432\u043e\u0434\u0430 C\/C++, \u043d\u0435 \u043e\u0441\u043e\u0431\u043e \u043d\u0430\u043f\u0440\u044f\u0433\u0430\u044f\u0441\u044c | ProHoster","og:description":"\u041f\u0440\u0435\u0434\u0438\u0441\u043b\u043e\u0432\u0438\u0435 \u0415\u0441\u0442\u044c.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-03-17T21:00:00+00:00","article:modified_time":"2020-02-18T10:44:56+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"42134","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 07:22:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:05:22","updated":"2026-01-22 07:22:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/42134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=42134"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/42134\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=42134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=42134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=42134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}