{"id":42134,"date":"2019-03-18T00:00:00","date_gmt":"2019-03-17T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas"},"modified":"2020-02-18T13:44:56","modified_gmt":"2020-02-18T10:44:56","slug":"uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","status":"publish","type":"post","link":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","title":{"rendered":"Accelerering af fil I\/O C\/C++, uden at anstrenge sig for meget","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Accelerering<\/p>\n

Forord<\/h4>\n

\nDer findes et simpelt og meget nyttigt v\u00e6rkt\u00f8j - BDelta<\/a><\/noindex>, og det viser sig, at det har v\u00e6ret en del af vores produktionsproces i meget lang tid (selv om jeg ikke kunne installere en version, var det helt sikkert ikke den seneste tilg\u00e6ngelige). Vi bruger det til dets prim\u00e6re form\u00e5l - at opbygge bin\u00e6re patches. N\u00e5r man ser p\u00e5, hvad der er i depotet, bliver man lidt trist: det er i bund og grund blevet forkastet for l\u00e6nge siden, og meget af det er st\u00e6rkt for\u00e6ldet (engang lavede min tidligere kollega nogle \u00e6ndringer der, men det var for l\u00e6nge siden). Generelt besluttede jeg at genoplive dette: jeg forkede, fjernede det, jeg ikke planl\u00e6gger at bruge, og flyttede projektet til cmake<\/a><\/noindex>, inline-de 'hot' mikrofunktioner, fjernede store arrays fra stacken (og variable l\u00e6ngde arrays, som tydeligt g\u00f8r mig irriteret), og k\u00f8rte profilen igen - og fandt ud af, at omkring 40% af tiden blev brugt p\u00e5 fwrite<\/a><\/noindex>\u2026
\n<\/a><\/noindex><\/p>\n

Hvad s\u00e5 med fwrite?<\/h4>\n

\nI denne kode kaldes fwrite (i mit specifikke testtilf\u00e6lde: opbygning af en patch mellem to n\u00e6rtst\u00e5ende 300 MB filer, inddata helt i hukommelsen) millioner af gange med en lille buffer. Det er \u00e5benlyst, at denne ting vil v\u00e6re en flaskehals, og derfor \u00f8nsker man at p\u00e5virke dette uden at skulle implementere forskellige datakilder eller asynkron I\/O lige nu; man ville finde en enklere l\u00f8sning. Det f\u00f8rste, der kom i tankerne - at \u00f8ge bufferst\u00f8rrelsen<\/p>\n

setvbuf(file, nullptr, _IOFBF, 64* 1024)<\/code><\/pre>\n
\nmen jeg fik ikke v\u00e6sentlige forbedringer (nu tog fwrite omkring 37% af tiden) - s\u00e5 sagen er ikke, at data skrives hyppigt til disken. Hvis man ser 'under motorhjelmen' af fwrite, kan man se, at der inde bagved sker lock\/unlock af FILE strukturen omtrent s\u00e5dan her (pseudo-kode, hele analysen blev udf\u00f8rt under Visual Studio 2017):<\/p>\n
\nsize_t fwrite (const void *buffer, size_t size, size_t count, FILE *stream)\n{\n   size_t retval = 0;\n   _lock_str(stream);   \/* lock stream *\/\n   __try\n   {\n      retval = _fwrite_nolock(buffer, size, count, stream);\n   }\n   __finally \n   {\n       _unlock_str(stream);   \/* unlock stream *\/\n   }\n   return retval;\n}\n<\/code><\/pre>\n
\nHvis man tror p\u00e5 profilen, tager _fwrite_nolock kun 6% af tiden, det resterende er overhead. I mit specifikke tilf\u00e6lde er tr\u00e5d-sikkerhed en \u00e5benlys overfl\u00f8dighed, som jeg vil ofre ved at erstatte kaldet til fwrite med _fwrite_nolock<\/a><\/noindex> \u2014 man skal ikke g\u00f8re det kompliceret, selv ikke med argumenter. Resultatet: denne simple manipulation har reduceret omkostningerne til at skrive resultaterne med flere gange, som i den oprindelige version udgjorde n\u00e6sten halvdelen af tidsforbruget. For \u00f8vrigt findes der en tilsvarende funktion i POSIX-verdenen \u2014 fwrite_unlocked<\/a><\/noindex>. Generelt set g\u00e6lder det samme for fread. P\u00e5 denne m\u00e5de kan man med et par #define f\u00e5 en ret s\u00e5 platform-uafh\u00e6ngig l\u00f8sning uden un\u00f8dvendige blokeringer i tilf\u00e6lde, hvor de ikke er n\u00f8dvendige (og det er ofte tilf\u00e6ldet).<\/p>\n
fwrite, _fwrite_nolock, setvbuf<\/h4>\n
\nLad os abstrahere fra det oprindelige projekt og fokusere p\u00e5 at teste et specifikt tilf\u00e6lde: skrivning af en stor fil (512 Mb) i ekstremt sm\u00e5 portioner \u2014 1 byte. Testsystem: AMD Ryzen 7 1700, 16 Gb RAM, HDD 7200 rpm med 64 Mb cache, Windows 10 1809, bin\u00e6ret blev bygget som 32-bit, optimeringer aktiveret, biblioteket blev statisk linket.<\/p>\n
Pr\u00f8ve til eksperimentet:<\/p>\n
\n#include <chrono>\n#include <cstdio>\n#include <inttypes.h>\n#include <memory>\n\n#ifdef _MSC_VER\n#define fwrite_unlocked _fwrite_nolock\n#endif\n\nusing namespace std::chrono;\n\nint main()\n{\n    std::unique_ptr<FILE, int(*)(FILE*)> file(fopen(\"test.bin\", \"wb\"), fclose);\n    if (!file)\n        return 1;\n\n    constexpr size_t TEST_BUFFER_SIZE = 256 * 1024;\n    if (setvbuf(file.get(), nullptr, _IOFBF, TEST_BUFFER_SIZE) != 0)\n        return 2;\n\n    auto start = steady_clock::now();\n    const uint8_t b = 77;\n    constexpr size_t TEST_FILE_SIZE = 512 * 1024 * 1024;\n    for (size_t i = 0; i < TEST_FILE_SIZE; ++i)\n        fwrite_unlocked(&b, 1, sizeof(b), file.get());\n\n    auto end = steady_clock::now();\n    auto interval = duration_cast<microseconds>(end - start);\n    printf(\"Time: %lldn\", interval.count());\n\n    return 0;\n}\n<\/code><\/pre>\n
\nSom variabler vil TEST_BUFFER_SIZE blive anvendt, og i et par tilf\u00e6lde vil vi erstatte fwrite_unlocked med fwrite. Lad os begynde med tilf\u00e6lde fwrite uden eksplicit at angive bufferst\u00f8rrelsen (vi kommenterer setvbuf og den relaterede kode): tid 27048906 \u00b5s, skrivehastighed \u2014 18.93 Mb\/s. Nu s\u00e6tter vi bufferst\u00f8rrelsen til 64 Kb: tid \u2014 25037111 \u00b5s, hastighed \u2014 20.44 Mb\/s. Lad os nu teste _fwrite_nolock uden at kalde setvbuf: 7262221 \u00b5s, hastighed \u2014 70.5 Mb\/s! <\/p>\n
Lad os eksperimentere videre med bufferst\u00f8rrelsen (setvbuf):<\/p>\n
\"Accelerering
\n
\nDataene blev opn\u00e5et som gennemsnit af 5 eksperimenter, jeg gad ikke at beregne fejlene. Personligt synes jeg, at 93 Mb\/s ved skrivning af 1 byte p\u00e5 en almindelig HDD \u2014 det er et ret godt resultat, man skal blot v\u00e6lge den optimale bufferst\u00f8rrelse (i mit tilf\u00e6lde 256 Kb \u2014 det er perfekt) og erstatte fwrite med _fwrite_nolock\/fwrite_unlocked (hvis man naturligvis ikke har brug for tr\u00e5dsikkerhed). 
\nDet samme g\u00e6lder for fread under lignende forhold. Da jeg ikke har en \"rigtig\" maskine med linux ved h\u00e5nden (enkortcomputerne t\u00e6ller ikke), besluttede jeg at udf\u00f8re et begr\u00e6nset eksperiment p\u00e5 en virtuel maskine (Hyper-V, OpenSUSE 15, GCC 8.3.1) \u2014 m\u00f8nstret er i princippet det samme: \"r\u00e5\" fwrite 20 Mb\/s, fwrite + buffer p\u00e5 256 Kb gav 23 Mb\/s, fwrite_unlocked med samme buffer \u2014 35 Mb\/s (bin\u00e6ret 64-bit, blev bygget g++ -o2 -s -static-libgcc -static-libstdc++ fwrite_test.cpp -o fwrite_test).<\/p>\n
Efterord<\/h4>\n
\nForm\u00e5let med denne artikel var at beskrive en simpel og effektiv metode, der i mange tilf\u00e6lde kan anvendes (jeg har aldrig arbejdet med funktionerne _fwrite_nolock\/fwrite_unlocked, de er ikke s\u00e5 popul\u00e6re \u2013 men det burde de v\u00e6re). Jeg g\u00f8r ikke krav p\u00e5 nyhedsv\u00e6rdien af materialet, men jeg h\u00e5ber, at artiklen vil v\u00e6re nyttig for samfundet.<\/p>\n
Kilde: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041f\u0440\u0435\u0434\u0438\u0441\u043b\u043e\u0432\u0438\u0435 \u0415\u0441\u0442\u044c \u043d\u0430 \u0441\u0432\u0435\u0442\u0435 \u0442\u0430\u043a\u0430\u044f \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0438 \u043e\u0447\u0435\u043d\u044c \u043f\u043e\u043b\u0435\u0437\u043d\u0430\u044f \u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u2014 BDelta, \u0438 \u0442\u0430\u043a \u0432\u044b\u0448\u043b\u043e, \u0447\u0442\u043e \u043e\u043d\u0430 \u043e\u0447\u0435\u043d\u044c \u0434\u0430\u0432\u043d\u043e \u0443\u043a\u043e\u0440\u0435\u043d\u0438\u043b\u0430\u0441\u044c \u0432 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u043e\u043c \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u0435 (\u043f\u0440\u0430\u0432\u0434\u0430 \u0435\u0451 \u0432\u0435\u0440\u0441\u0438\u044e \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c \u043d\u0435 \u0443\u0434\u0430\u043b\u043e\u0441\u044c, \u043d\u043e \u043e\u043d\u0430 \u0442\u043e\u0447\u043d\u043e \u0431\u044b\u043b\u0430 \u043d\u0435 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0435\u0439 \u0434\u043e\u0441\u0442\u0443\u043f\u043d\u043e\u0439). \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c \u0435\u0451 \u043f\u043e \u043f\u0440\u044f\u043c\u043e\u043c\u0443 \u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u0438\u044e \u2014 \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u0435 \u0431\u0438\u043d\u0430\u0440\u043d\u044b\u0445 \u043f\u0430\u0442\u0447\u0435\u0439. \u0415\u0441\u043b\u0438 \u0432\u0437\u0433\u043b\u044f\u043d\u0443\u0442\u044c, \u0447\u0442\u043e \u0442\u0430\u043c \u0432 \u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u0438, \u2014 \u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0441\u044f \u0441\u043b\u0435\u0433\u043a\u0430 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-42134","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Accelerering af filindl\u00e6sning i C\/C++, uden s\u00e6rlig anstrengelse | ProHoster","description":"Forord","canonical_url":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"da_DK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0423\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u0435 \u0444\u0430\u0439\u043b\u043e\u0432\u043e\u0433\u043e \u0432\u0432\u043e\u0434\u0430-\u0432\u044b\u0432\u043e\u0434\u0430 C\/C++, \u043d\u0435 \u043e\u0441\u043e\u0431\u043e \u043d\u0430\u043f\u0440\u044f\u0433\u0430\u044f\u0441\u044c | ProHoster","og:description":"\u041f\u0440\u0435\u0434\u0438\u0441\u043b\u043e\u0432\u0438\u0435 \u0415\u0441\u0442\u044c.","og:url":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/uskorenie-fajlovogo-vvoda-vyvoda-c-c-ne-osobo-napryagayas","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-03-17T21:00:00+00:00","article:modified_time":"2020-02-18T10:44:56+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"42134","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 07:22:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 00:05:22","updated":"2026-01-22 07:22:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/42134","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/comments?post=42134"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/42134\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/media?parent=42134"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/categories?post=42134"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/tags?post=42134"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}