{"id":113018,"date":"2024-01-20T23:28:04","date_gmt":"2024-01-20T21:28:06","guid":{"rendered":"https:\/\/prohoster.info\/blog\/novosti-interneta\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu"},"modified":"2024-01-20T23:28:04","modified_gmt":"2024-01-20T21:28:06","slug":"opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","status":"publish","type":"post","link":"https:\/\/prohoster.info\/et\/blog\/news\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","title":{"rendered":"Cephi andmehoidla loomise kogemus tebibaitides sekundis","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Clyso insener viis kokkuv\u00f5tte Cephi baasil p\u00fcsim\u00e4lu klastrite loomise kogemusest, mille l\u00e4bilaskev\u00f5ime \u00fcletab tebibaiti sekundis. T\u00f5detakse, et see on esimene Cephile tuginev klaster, mis suutis selliseid n\u00e4itajaid saavutada, kuid enne esitatud tulemuse saavutamist pidi inseneridel \u00fcletama mitmeid keerulisi takistusi. <\/p>\n

N\u00e4iteks j\u00f5udluse suurendamiseks 10-20% piisab, kui aktiveerida serverite BIOS-i energias\u00e4\u00e4sture\u017eiimis ainult maksimaalse j\u00f5udluse re\u017eiim ja v\u00e4lja l\u00fclitada c-state (c-state muudab energias\u00e4\u00e4stu parameetreid vastavalt koormusele, mis m\u00f5jutab Cephi). Samuti selgus, et NVMe m\u00e4luseadmete kasutamisel kulutab Linuxi s\u00fcdamik olulise aja spin-lockide t\u00f6\u00f6tlemiseks IOMMU kaardistuse uuendamise protsessis. IOMMU v\u00e4ljal\u00fclitamine s\u00fcdamikus t\u00f5i kaasa m\u00e4rgatava j\u00f5udluse kasvu blokktestides, mille suurus oli 4MB. <\/p>\n

Kuna selle t\u00f5ttu IOMMU v\u00e4ljaj\u00e4tmine probleem ei lahendanud 4KB plokkide juhuslikku kirjutamist sooritatavatest j\u00f5udluse langust. Uurides p\u00f5hjus, sattusid insenerid Gentoo ja Ubuntu projektide Ceph'i koostamisstsenaariumite parandustega, mis sisaldasid koostamist valikuga RelWithDebInfo, kuna selle puhul rakendati GCC-s optimeerimisre\u017eiimi '-O2', millega Ceph'i j\u00f5udlus m\u00e4rgatavalt paranes. J\u00f5udluse langusele aitas kaasa ka TCMalloc'i teegi kasutamine. Kompilaatoriflagide muutmine ja TCMalloc'i l\u00f5petamine viis pakendamise (compaction) aja v\u00e4henemiseni kolmandiku v\u00f5rra ja 4K plokkide juhuslike kirjutamise j\u00f5udluse kahekordistumiseni. L\u00f5ppkokkuv\u00f5ttes tehti ka Reef RocksDB ja paigutusgruppide (PG) seadetesse t\u00e4iendav optimeerimine. <\/p>\n

Klaster on moodustatud 68 s\u00f5lmest, mis p\u00f5hinevad serverid<\/a> Dell PowerEdge R6615 koos CPU AMD EPYC 9454P 48C\/96T. Igal s\u00f5lmel on 10 NVMe-andme\u00fchikut Dell 15.36TB, kaks Ethernet-adapterit 100GbE Mellanox ConnectX-6 ja 192 GB RAM-i. Tarkvara p\u00f5hineb Ubuntu 20.04.6-l ja Ceph 17.2.7-l. 63-s\u00f5lmelises klastris on seadistatud 630 OSD (Object Storage Daemon, taustprotsess, mis haldab andmete salvestamist kohalikus salvestuses, \u00fcks OSD iga NVMe-andme\u00fchiku kohta), kolm MON-protsessi (monitor, j\u00e4lgib klastri olekut) ja \u00fcks MGR-protsess (Manager, halduss\u00fcsteem). Salvestusmaht on 8.2 PB. <\/p>\n

Sekventiaalsete 4M plokkide lugemise operatsioonide l\u00e4bilaskev\u00f5ime oli 1025 GiB\/s, kirjutamise - 270 GiB\/s. Juhuslikul 4KB plokkide lugemisel oli j\u00f5udlus 25,5 miljonit lugemisoperatsiooni sekundis ja 4,9 miljonit kirjutamise osas. Kr\u00fcpteerimise lubamine v\u00e4hendas lugemise l\u00e4bilaskev\u00f5imet ligikaudu 750 GiB\/s. Viga parandavate koodide EC62 kasutamisel oli l\u00e4bilaskev\u00f5ime 547 GiB\/s lugemisel ja 387 GiB\/s kirjutamisel (kirjutamise kiirus oli k\u00f5rgem kui ilma paranduskoodideta), ning juhuslikul juurdep\u00e4\u00e4sul 3,4M IOPS lugemisel ja 936K IOPS kirjutamisel.

\"Cephi<\/center>
\"Cephi<\/center> <\/p>\n

Lisaks v\u00f5ib m\u00e4rkida, et septembris saavutati sarnane ribalaius tebibaiti sekundis CERN-i eksabaidi salvestusklastri puhul, mis on rakendatud avatud jaotatud salvestuss\u00fcsteemi EOS p\u00f5hjal, mis on \u00fcles ehitatud XRootD protokolli p\u00f5hjal.
\n
Allikas: opennet.ru<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u0418\u043d\u0436\u0435\u043d\u0435\u0440 \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Clyso \u043e\u0431\u043e\u0431\u0449\u0438\u043b \u043e\u043f\u044b\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043d\u0430 \u0431\u0430\u0437\u0435 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0439 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b Ceph \u0441 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c\u044e, \u043f\u0440\u0435\u0432\u044b\u0448\u0430\u044e\u0449\u0435\u0439 \u0442\u0435\u0431\u0438\u0431\u0430\u0439\u0442 \u0432 \u0441\u0435\u043a\u0443\u043d\u0434\u0443. \u041e\u0442\u043c\u0435\u0447\u0430\u0435\u0442\u0441\u044f, \u0447\u0442\u043e \u044d\u0442\u043e \u043f\u0435\u0440\u0432\u044b\u0439 \u043a\u043b\u0430\u0441\u0442\u0435\u0440 \u043d\u0430 \u0431\u0430\u0437\u0435 Ceph, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0441\u043c\u043e\u0433 \u0434\u043e\u0441\u0442\u0438\u0433\u043d\u0443\u0442\u044c \u043f\u043e\u0434\u043e\u0431\u043d\u043e\u0433\u043e \u043f\u043e\u043a\u0430\u0437\u0430\u0442\u0435\u043b\u044f, \u043d\u043e \u043f\u0435\u0440\u0435\u0434 \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u0435\u043c \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u0435\u043d\u043d\u043e\u0433\u043e \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0430 \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u0430\u043c \u043f\u043e\u0442\u0440\u0435\u0431\u043e\u0432\u0430\u043b\u043e\u0441\u044c \u043f\u0440\u0435\u043e\u0434\u043e\u043b\u0435\u0442\u044c \u0441\u0435\u0440\u0438\u044e \u043d\u0435 \u043e\u0447\u0435\u0432\u0438\u0434\u043d\u044b\u0445 \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0445 \u043a\u0430\u043c\u043d\u0435\u0439. \u041d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0434\u043b\u044f \u043f\u043e\u0432\u044b\u0448\u0435\u043d\u0438\u044f \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u043d\u0430 10-20% \u0431\u044b\u043b\u043e […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":113019,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-113018","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Ceph salvestuslahenduse loomise kogemus, mille ribalaius on tebibaiti sekundis | ProHoster","description":"Clyso insener kokkuv\u00f5ttis kogemuse andmeklastri loomisel, mis p\u00f5hineb vigadetuks j\u00e4\u00e4val jagatud s\u00fcsteemil Ceph, mille ribalaius \u00fcletab tebibaiti sekundis.","canonical_url":"https:\/\/prohoster.info\/et\/blog\/news\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"et_EE","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u044b\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0430 Ceph c \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c\u044e \u0442\u0435\u0431\u0438\u0431\u0430\u0439\u0442 \u0432 \u0441\u0435\u043a\u0443\u043d\u0434\u0443 | ProHoster","og:description":"\u0418\u043d\u0436\u0435\u043d\u0435\u0440 \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Clyso \u043e\u0431\u043e\u0431\u0449\u0438\u043b \u043e\u043f\u044b\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043d\u0430 \u0431\u0430\u0437\u0435 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0439 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b Ceph \u0441 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c\u044e, \u043f\u0440\u0435\u0432\u044b\u0448\u0430\u044e\u0449\u0435\u0439 \u0442\u0435\u0431\u0438\u0431\u0430\u0439\u0442 \u0432 \u0441\u0435\u043a\u0443\u043d\u0434\u0443.","og:url":"https:\/\/prohoster.info\/et\/blog\/news\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2024-01-20T21:28:06+00:00","article:modified_time":"2024-01-20T21:28:06+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"113018","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2026-02-09 16:50:41","updated":"2026-02-09 16:50:41","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/113018","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/comments?post=113018"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/113018\/revisions"}],"predecessor-version":[{"id":158784,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/113018\/revisions\/158784"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media\/113019"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media?parent=113018"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/categories?post=113018"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/tags?post=113018"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}