{"id":42134,"date":"2019-03-18T00:00:00","date_gmt":"2019-03-17T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas"},"modified":"2020-02-18T13:44:56","modified_gmt":"2020-02-18T10:44:56","slug":"uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","status":"publish","type":"post","link":"https:\/\/prohoster.info\/pt\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","title":{"rendered":"Acelera\u00e7\u00e3o da entrada e sa\u00edda de arquivos C\/C++, sem muito esfor\u00e7o","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Acelera\u00e7\u00e3o<\/p>\n

Pref\u00e1cio<\/h4>\n

\nExiste uma ferramenta t\u00e3o simples e muito \u00fatil no mundo \u2014 BDelta<\/a><\/noindex>, e aconteceu que ela se firmou em nosso processo produtivo h\u00e1 muito tempo (apesar de n\u00e3o ter conseguido instalar a vers\u00e3o dela, com certeza n\u00e3o era a mais recente dispon\u00edvel). Usamos para seu prop\u00f3sito original \u2014 a constru\u00e7\u00e3o de patches bin\u00e1rios. Ao olhar o que h\u00e1 no reposit\u00f3rio, d\u00e1 uma leve tristeza: basicamente, ele foi abandonado h\u00e1 tempos e muitas coisas l\u00e1 est\u00e3o bastante desatualizadas (uma vez, meu ex-colega fez algumas corre\u00e7\u00f5es, mas isso faz muito tempo). Em geral, decidi reviver este projeto: fiz um fork, removi o que n\u00e3o pretendo usar, e migrei o projeto para cmake<\/a><\/noindex>, inlined as microfun\u00e7\u00f5es \u201cquentes\u201d, removi grandes arrays da pilha (e arrays de comprimento vari\u00e1vel, que me deixaram bastante irritado), passei o profiler mais uma vez \u2014 e descobri que cerca de 40% do tempo \u00e9 gasto em fwrite<\/a><\/noindex>\u2026
\n<\/a><\/noindex><\/p>\n

E quanto ao fwrite?<\/h4>\n

\nNeste c\u00f3digo, fwrite (no meu caso espec\u00edfico: construindo um patch entre arquivos de cerca de 300 MB, com dados totalmente em mem\u00f3ria) \u00e9 chamado milh\u00f5es de vezes com um buffer pequeno. \u00c9 \u00f3bvio que essa situa\u00e7\u00e3o vai ser um gargalo, e, portanto, gostaria de impactar isso de alguma forma sem grandes complica\u00e7\u00f5es. N\u00e3o tenho interesse em implementar diversos tipos de fontes de dados, ou I\/O ass\u00edncrono, ent\u00e3o queria encontrar uma solu\u00e7\u00e3o mais simples. Primeiro que me veio \u00e0 mente \u2014 aumentar o tamanho do buffer.<\/p>\n

setvbuf(file, nullptr, _IOFBF, 64 * 1024)<\/code><\/pre>\n
\nmas n\u00e3o obtive uma melhoria significativa no resultado (agora, fwrite consumia cerca de 37% do tempo) \u2014 ent\u00e3o a quest\u00e3o n\u00e3o era a grava\u00e7\u00e3o frequente de dados no disco. Olhando \u201cdebaixo do cap\u00f4\u201d do fwrite, podemos ver que internamente ocorre um lock\/unlock da estrutura FILE mais ou menos assim (pseudoc\u00f3digo, toda a an\u00e1lise foi realizada no Visual Studio 2017):<\/p>\n
\nsize_t fwrite (const void *buffer, size_t size, size_t count, FILE *stream)\n{\n   size_t retval = 0;\n   _lock_str(stream);   \/* lock stream *\/\n   __try\n   {\n      retval = _fwrite_nolock(buffer, size, count, stream);\n   }\n   __finally \n   {\n       _unlock_str(stream);   \/* unlock stream *\/\n   }\n   return retval;\n}\n<\/code><\/pre>\n
\nSe confiar no profiler, apenas 6% do tempo \u00e9 gasto em _fwrite_nolock, o restante \u00e9 overhead. No meu caso espec\u00edfico, a seguran\u00e7a de thread \u00e9 um excesso \u00f3bvio, e vou abdic\u00e1-la, substituindo a chamada do fwrite por _fwrite_nolock<\/a><\/noindex> \u2014 nem \u00e9 preciso complicar com argumentos. No total: essa simples manipula\u00e7\u00e3o reduziu drasticamente os custos de grava\u00e7\u00e3o do resultado, que na vers\u00e3o original representavam quase metade do tempo total gasto. A prop\u00f3sito, no mundo POSIX existe uma fun\u00e7\u00e3o an\u00e1loga \u2014 fwrite_unlocked<\/a><\/noindex>. De modo geral, o mesmo se aplica a fread. Dessa forma, com um par de #define, \u00e9 poss\u00edvel obter uma solu\u00e7\u00e3o multiplataforma bastante eficiente, sem bloqueios desnecess\u00e1rios quando n\u00e3o s\u00e3o necess\u00e1rios (o que acontece com bastante frequ\u00eancia).<\/p>\n
fwrite, _fwrite_nolock, setvbuf<\/h4>\n
\nVamos nos abstrair do projeto original e nos concentrar na testagem de um caso espec\u00edfico: a grava\u00e7\u00e3o de um grande arquivo (512 MB) em por\u00e7\u00f5es extremamente pequenas \u2014 de 1 byte. Sistema de teste: AMD Ryzen 7 1700, 16 GB de RAM, HDD 7200 rpm com 64 MB de cache, Windows 10 1809, bin\u00e1rio constru\u00eddo em 32 bits, otimiza\u00e7\u00f5es ativadas, biblioteca est\u00e1ticamente linkada.<\/p>\n
Amostra para realiza\u00e7\u00e3o do experimento:<\/p>\n
\n#include <chrono>\n#include <cstdio>\n#include <inttypes.h>\n#include <memory>\n\n#ifdef _MSC_VER\n#define fwrite_unlocked _fwrite_nolock\n#endif\n\nusing namespace std::chrono;\n\nint main()\n{\n    std::unique_ptr<FILE, int(*)(FILE*)> file(fopen(\"test.bin\", \"wb\"), fclose);\n    if (!file)\n        return 1;\n\n    constexpr size_t TEST_BUFFER_SIZE = 256 * 1024;\n    if (setvbuf(file.get(), nullptr, _IOFBF, TEST_BUFFER_SIZE) != 0)\n        return 2;\n\n    auto start = steady_clock::now();\n    const uint8_t b = 77;\n    constexpr size_t TEST_FILE_SIZE = 512 * 1024 * 1024;\n    for (size_t i = 0; i < TEST_FILE_SIZE; ++i)\n        fwrite_unlocked(&b, 1, sizeof(b), file.get());\n\n    auto end = steady_clock::now();\n    auto interval = duration_cast<microseconds>(end - start);\n    printf(\"Time: %lldn\", interval.count());\n\n    return 0;\n}\n<\/code><\/pre>\n
\nOs vari\u00e1veis ser\u00e3o TEST_BUFFER_SIZE, e, para um par de casos, substituiremos fwrite_unlocked por fwrite. Comecemos com o caso fwrite sem defini\u00e7\u00e3o expl\u00edcita do tamanho do buffer (comentaremos setvbuf e o c\u00f3digo relacionado): tempo 27048906 \u00b5s, velocidade de grava\u00e7\u00e3o \u2014 18.93 MB\/s. Agora, definimos o tamanho do buffer em 64 KB: tempo \u2014 25037111 \u00b5s, velocidade \u2014 20.44 MB\/s. Agora, testaremos o funcionamento de _fwrite_nolock sem a chamada de setvbuf: 7262221 \u00b5s, velocidade \u2014 70.5 MB\/s! <\/p>\n
Em seguida, vamos experimentar com o tamanho do buffer (setvbuf):<\/p>\n
\"Acelera\u00e7\u00e3o
\n
\nOs dados foram obtidos pela m\u00e9dia de 5 experimentos, n\u00e3o me dei ao trabalho de calcular as margens de erro. Na minha opini\u00e3o, 93 MB\/s ao gravar 1 byte em um HDD comum \u00e9 um resultado muito bom, s\u00f3 \u00e9 preciso escolher o tamanho \u00f3timo do buffer (no meu caso, 256 KB \u2014 \u00e9 o ideal) e substituir fwrite por _fwrite_nolock\/fwrite_unlocked (caso a seguran\u00e7a de thread n\u00e3o seja necess\u00e1ria, claro). 
\nO mesmo se aplica ao fread em condi\u00e7\u00f5es semelhantes. Como n\u00e3o tenho uma m\u00e1quina 'f\u00edsica' com linux \u00e0 m\u00e3o (placas \u00fanicas n\u00e3o contam), decidi realizar um experimento limitado em uma m\u00e1quina virtual (Hyper-V, OpenSUSE 15, GCC 8.3.1) \u2014 a regularidade, em princ\u00edpio, \u00e9 a mesma: fwrite puro 20 MB\/s, fwrite + buffer de 256 KB forneceu 23 MB\/s, fwrite_unlocked com o mesmo buffer \u2014 35 MB\/s (bin\u00e1rio de 64 bits, constru\u00eddo com g++ -o2 -s -static-libgcc -static-libstdc++ fwrite_test.cpp -o fwrite_test).<\/p>\n
Ep\u00edlogo<\/h4>\n
\nO objetivo deste artigo \u00e9 descrever uma abordagem simples e eficaz que pode ser \u00fatil em muitos casos (n\u00e3o havia me deparado antes com as fun\u00e7\u00f5es _fwrite_nolock\/fwrite_unlocked, que n\u00e3o s\u00e3o muito populares \u2014 e isso \u00e9 uma pena). N\u00e3o pretendo reivindicar a novidade do material, mas espero que o artigo seja \u00fatil para a comunidade.<\/p>\n
Fonte: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041f\u0440\u0435\u0434\u0438\u0441\u043b\u043e\u0432\u0438\u0435 \u0415\u0441\u0442\u044c \u043d\u0430 \u0441\u0432\u0435\u0442\u0435 \u0442\u0430\u043a\u0430\u044f \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0438 \u043e\u0447\u0435\u043d\u044c \u043f\u043e\u043b\u0435\u0437\u043d\u0430\u044f \u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u2014 BDelta, \u0438 \u0442\u0430\u043a \u0432\u044b\u0448\u043b\u043e, \u0447\u0442\u043e \u043e\u043d\u0430 \u043e\u0447\u0435\u043d\u044c \u0434\u0430\u0432\u043d\u043e \u0443\u043a\u043e\u0440\u0435\u043d\u0438\u043b\u0430\u0441\u044c \u0432 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435 (\u043f\u0440\u0430\u0432\u0434\u0430 \u0435\u0451 \u0432\u0435\u0440\u0441\u0438\u044e \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c \u043d\u0435 \u0443\u0434\u0430\u043b\u043e\u0441\u044c, \u043d\u043e \u043e\u043d\u0430 \u0442\u043e\u0447\u043d\u043e \u0431\u044b\u043b\u0430 \u043d\u0435 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0435\u0439 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0439). \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u0435\u0451 \u043f\u043e \u043f\u0440\u044f\u043c\u043e\u043c\u0443 \u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044e \u2014 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u0435 \u0431\u0438\u043d\u0430\u0440\u043d\u044b\u0445 \u043f\u0430\u0442\u0447\u0435\u0439. \u0415\u0441\u043b\u0438 \u0432\u0437\u0433\u043b\u044f\u043d\u0443\u0442\u044c, \u0447\u0442\u043e \u0442\u0430\u043c \u0432 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438, \u2014 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u0441\u043b\u0435\u0433\u043a\u0430 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-42134","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Acelera\u00e7\u00e3o de entrada e sa\u00edda de arquivos em C\/C++, sem muito esfor\u00e7o | ProHoster","description":"Pref\u00e1cio.","canonical_url":"https:\/\/prohoster.info\/pt\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"pt_PT","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0423\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u0435 \u0444\u0430\u0439\u043b\u043e\u0432\u043e\u0433\u043e \u0432\u0432\u043e\u0434\u0430-\u0432\u044b\u0432\u043e\u0434\u0430 C\/C++, \u043d\u0435 \u043e\u0441\u043e\u0431\u043e \u043d\u0430\u043f\u0440\u044f\u0433\u0430\u044f\u0441\u044c | ProHoster","og:description":"\u041f\u0440\u0435\u0434\u0438\u0441\u043b\u043e\u0432\u0438\u0435 \u0415\u0441\u0442\u044c.","og:url":"https:\/\/prohoster.info\/pt\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-03-17T21:00:00+00:00","article:modified_time":"2020-02-18T10:44:56+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"42134","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 07:22:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:05:22","updated":"2026-01-22 07:22:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/posts\/42134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/comments?post=42134"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/posts\/42134\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/media?parent=42134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/categories?post=42134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/pt\/wp-json\/wp\/v2\/tags?post=42134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}