{"id":113018,"date":"2024-01-20T23:28:04","date_gmt":"2024-01-20T21:28:06","guid":{"rendered":"https:\/\/prohoster.info\/blog\/novosti-interneta\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu"},"modified":"2024-01-20T23:28:04","modified_gmt":"2024-01-20T21:28:06","slug":"opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/news\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","title":{"rendered":"Zku\u0161enost s vytv\u00e1\u0159en\u00edm \u00falo\u017ei\u0161t\u011b Ceph s propustnost\u00ed tebibit\u016f za sekundu","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

In\u017een\u00fdr z firmy Clyso shrnul zku\u0161enosti s vytvo\u0159en\u00edm \u00falo\u017ei\u0161t\u011b klastru na b\u00e1zi redundantn\u00edho distribuovan\u00e9ho syst\u00e9mu Ceph s propustnost\u00ed p\u0159esahuj\u00edc\u00ed tebibyte za sekundu. Bylo poznamen\u00e1no, \u017ee toto je prvn\u00ed klastra na b\u00e1zi Ceph, kter\u00e1 dok\u00e1zala dos\u00e1hnout takov\u00e9ho ukazatele, ale p\u0159ed dosa\u017een\u00edm tohoto v\u00fdsledku bylo pot\u0159eba p\u0159ekonat \u0159adu nep\u0159edv\u00eddan\u00fdch p\u0159ek\u00e1\u017eek. <\/p>\n

Nap\u0159\u00edklad pro zv\u00fd\u0161en\u00ed v\u00fdkonu o 10-20 % sta\u010dilo na serverech v nastaven\u00ed \u0161et\u0159en\u00ed energi\u00ed v BIOSu nastavit re\u017eim maxim\u00e1ln\u00edho v\u00fdkonu a vypnout c-state (c-state m\u011bn\u00ed parametry \u0161et\u0159en\u00ed energie v z\u00e1vislosti na zat\u00ed\u017een\u00ed, co\u017e se odr\u00e1\u017e\u00ed v Ceph). Tak\u00e9 se uk\u00e1zalo, \u017ee p\u0159i pou\u017eit\u00ed disk\u016f NVMe j\u00e1dro Linuxu tr\u00e1v\u00ed v\u00fdznamnou \u010d\u00e1st \u010dasu zpracov\u00e1n\u00edm spin-lock\u016f b\u011bhem aktualizace mapov\u00e1n\u00ed IOMMU. Vypnut\u00ed IOMMU v j\u00e1dru vedlo k v\u00fdrazn\u00e9mu zv\u00fd\u0161en\u00ed v\u00fdkonu ve testech z\u00e1pisu a \u010dten\u00ed blok\u016f o velikosti 4MB. <\/p>\n

Vypnut\u00ed IOMMU v\u0161ak nevy\u0159e\u0161ilo probl\u00e9m se sni\u017eov\u00e1n\u00edm v\u00fdkonu p\u0159i n\u00e1hodn\u00e9m z\u00e1pisu blok\u016f o velikosti 4 KB. P\u0159i zkoum\u00e1n\u00ed p\u0159\u00ed\u010diny in\u017een\u00fd\u0159i narazili na opravy ve skriptech pro sestaven\u00ed Ceph z projekt\u016f Gentoo a Ubuntu, kter\u00e9 zahrnovaly sestaven\u00ed s parametrem RelWithDebInfo, proto\u017ee s n\u00edm byl v GCC pou\u017eit optimaliza\u010dn\u00ed re\u017eim \u201a-O2\u2018, p\u0159i kter\u00e9m v\u00fdrazn\u011b vzrostl v\u00fdkon Ceph. K poklesu v\u00fdkonu tak\u00e9 p\u0159isp\u011bla kompilace s knihovnou TCMalloc. Zm\u011bna p\u0159ep\u00edna\u010d\u016f kompilace a p\u0159eru\u0161en\u00ed pou\u017e\u00edv\u00e1n\u00ed TCMalloc vedly k trojn\u00e1sobn\u00e9mu sn\u00ed\u017een\u00ed doby balen\u00ed (compaction) a dvojn\u00e1sobn\u00e9mu zv\u00fd\u0161en\u00ed v\u00fdkonu n\u00e1hodn\u00fdch z\u00e1pisov\u00fdch operac\u00ed bloky 4K. Na z\u00e1v\u011br byla nav\u00edc optimalizov\u00e1na nastaven\u00ed Reef RocksDB a skupiny um\u00edst\u011bn\u00ed (PG). <\/p>\n

Klastr je tvo\u0159en ze 68 uzl\u016f na b\u00e1zi servery<\/a> Dell PowerEdge R6615 s CPU AMD EPYC 9454P 48C\/96T. Ka\u017ed\u00fd uzel obsahuje 10 NVMe disk\u016f Dell 15.36TB, dva ethernetov\u00e9 adapt\u00e9ry 100GbE Mellanox ConnectX-6 a 192 GB RAM. Software je zalo\u017een na Ubuntu 20.04.6 a Ceph 17.2.7. V klastru na 63 uzlech je nasazeno 630 OSD (Object Storage Daemon, proces na pozad\u00ed, kter\u00fd spravuje ukl\u00e1d\u00e1n\u00ed dat v m\u00edstn\u00edm \u00falo\u017ei\u0161ti, po jednom OSD na NVMe disk), t\u0159i procesy MON (monitor, sleduje stav klastra) a jeden proces MGR (Manager, spravuj\u00edc\u00ed slu\u017eba). Velikost \u00falo\u017ei\u0161t\u011b je 8.2 PB. <\/p>\n

Propustnost p\u0159i sekven\u010dn\u00edch operac\u00edch \u010dten\u00ed blok\u016f 4M \u010dinila 1025 GiB\/s, z\u00e1pisu \u2014 270 GiB\/s. P\u0159i n\u00e1hodn\u00e9m \u010dten\u00ed blok\u016f 4KB v\u00fdkon \u010dinil 25,5 milionu operac\u00ed \u010dten\u00ed za sekundu a 4,9 milionu p\u0159i z\u00e1pisu. Aktivace \u0161ifrov\u00e1n\u00ed sn\u00ed\u017eila propustnost p\u0159i \u010dten\u00ed p\u0159ibli\u017en\u011b na 750 GiB\/s. P\u0159i pou\u017eit\u00ed k\u00f3d\u016f pro opravu chyb EC62 \u010dinila propustnost 547 GiB\/s p\u0159i \u010dten\u00ed a 387 GiB\/s p\u0159i z\u00e1pisu (rychlost z\u00e1pisu byla vy\u0161\u0161\u00ed ne\u017e bez k\u00f3d\u016f pro opravu), a p\u0159i n\u00e1hodn\u00e9m p\u0159\u00edstupu 3,4M IOPS p\u0159i \u010dten\u00ed a 936K IOPS p\u0159i z\u00e1pisu.

\"Zku\u0161enost<\/center>
\"Zku\u0161enost<\/center> <\/p>\n

D\u00e1le je t\u0159eba poznamenat, \u017ee v z\u00e1\u0159\u00ed byl podobn\u00fd miln\u00edk p\u0159enosov\u00e9 kapacity v tebibajtech za sekundu dosa\u017een v exabajtov\u00e9m \u00falo\u017ei\u0161ti CERN, realizovan\u00e9m na z\u00e1klad\u011b otev\u0159en\u00e9ho distribuovan\u00e9ho \u00falo\u017ei\u0161t\u011b EOS, postaven\u00e9ho na protokolu XRootD.
\n
Zdroj: opennet.ru<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u0418\u043d\u0436\u0435\u043d\u0435\u0440 \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Clyso \u043e\u0431\u043e\u0431\u0449\u0438\u043b \u043e\u043f\u044b\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043d\u0430 \u0431\u0430\u0437\u0435 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0439 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b Ceph \u0441 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c\u044e, \u043f\u0440\u0435\u0432\u044b\u0448\u0430\u044e\u0449\u0435\u0439 \u0442\u0435\u0431\u0438\u0431\u0430\u0439\u0442 \u0432 \u0441\u0435\u043a\u0443\u043d\u0434\u0443. \u041e\u0442\u043c\u0435\u0447\u0430\u0435\u0442\u0441\u044f, \u0447\u0442\u043e \u044d\u0442\u043e \u043f\u0435\u0440\u0432\u044b\u0439 \u043a\u043b\u0430\u0441\u0442\u0435\u0440 \u043d\u0430 \u0431\u0430\u0437\u0435 Ceph, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0441\u043c\u043e\u0433 \u0434\u043e\u0441\u0442\u0438\u0433\u043d\u0443\u0442\u044c \u043f\u043e\u0434\u043e\u0431\u043d\u043e\u0433\u043e \u043f\u043e\u043a\u0430\u0437\u0430\u0442\u0435\u043b\u044f, \u043d\u043e \u043f\u0435\u0440\u0435\u0434 \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u0438\u0435\u043c \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u0435\u043d\u043d\u043e\u0433\u043e \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442\u0430 \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u0430\u043c \u043f\u043e\u0442\u0440\u0435\u0431\u043e\u0432\u0430\u043b\u043e\u0441\u044c \u043f\u0440\u0435\u043e\u0434\u043e\u043b\u0435\u0442\u044c \u0441\u0435\u0440\u0438\u044e \u043d\u0435 \u043e\u0447\u0435\u0432\u0438\u0434\u043d\u044b\u0445 \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0445 \u043a\u0430\u043c\u043d\u0435\u0439. \u041d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0434\u043b\u044f \u043f\u043e\u0432\u044b\u0448\u0435\u043d\u0438\u044f \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u043d\u0430 10-20% \u0431\u044b\u043b\u043e […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":113019,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-113018","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Zku\u0161enost s vytvo\u0159en\u00edm \u00falo\u017ei\u0161t\u011b Ceph s p\u0159enosovou kapacitou tebibajt za sekundu | ProHoster","description":"In\u017een\u00fdr z firmy Clyso shrnul zku\u0161enosti s vytv\u00e1\u0159en\u00edm \u00falo\u017ei\u0161t\u011b na b\u00e1zi odoln\u00e9ho distribuovan\u00e9ho syst\u00e9mu Ceph s p\u0159enosovou kapacitou p\u0159esahuj\u00edc\u00ed tebibajt za sekundu.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/news\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u044b\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0430 Ceph c \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c\u044e \u0442\u0435\u0431\u0438\u0431\u0430\u0439\u0442 \u0432 \u0441\u0435\u043a\u0443\u043d\u0434\u0443 | ProHoster","og:description":"\u0418\u043d\u0436\u0435\u043d\u0435\u0440 \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Clyso \u043e\u0431\u043e\u0431\u0449\u0438\u043b \u043e\u043f\u044b\u0442 \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u043d\u0430 \u0431\u0430\u0437\u0435 \u043e\u0442\u043a\u0430\u0437\u043e\u0443\u0441\u0442\u043e\u0439\u0447\u0438\u0432\u043e\u0439 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0451\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b Ceph \u0441 \u043f\u0440\u043e\u043f\u0443\u0441\u043a\u043d\u043e\u0439 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u044c\u044e, \u043f\u0440\u0435\u0432\u044b\u0448\u0430\u044e\u0449\u0435\u0439 \u0442\u0435\u0431\u0438\u0431\u0430\u0439\u0442 \u0432 \u0441\u0435\u043a\u0443\u043d\u0434\u0443.","og:url":"https:\/\/prohoster.info\/cs\/blog\/news\/opyt-sozdaniya-hranilishha-ceph-c-propusknoj-sposobnostyu-tebibajt-v-sekundu","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2024-01-20T21:28:06+00:00","article:modified_time":"2024-01-20T21:28:06+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"113018","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2026-02-09 16:50:41","updated":"2026-02-09 16:50:41","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/113018","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=113018"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/113018\/revisions"}],"predecessor-version":[{"id":158784,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/113018\/revisions\/158784"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/113019"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=113018"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=113018"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=113018"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}